2023-10-30T17:12:29.492 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-30T17:12:29.493 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201 --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201/orig.config.yaml 2023-10-30T17:12:29.535 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T17:12:29.617 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201 branch: wip-yuri10-testing-2023-10-27-0804 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7441201' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri10-testing-2023-10-27-0804 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: fee4af45ddecf08e014b4503e218408d9e5c5be2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: fee4af45ddecf08e014b4503e218408d9e5c5be2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2023-10-27-0804 sha1: fee4af45ddecf08e014b4503e218408d9e5c5be2 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 5679 sha1: fee4af45ddecf08e014b4503e218408d9e5c5be2 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2023-10-27-0804 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: fee4af45ddecf08e014b4503e218408d9e5c5be2 targets: smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEsA2T3Z7BhEScoWd2R+zeQ3a4mkh6Qpyg5hRdYQ+wKe/dOBtKMtrJRAdobB9gMKKCdoByietgpKaqqfGE15ufg= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKldg/iMffn8uQIfGj3LsSXEf/wPuI0S90sdsB603K8/X9pi5GwOwl1hilfS5DV06t4Vp5SxbwhF17bGKJ+zSg= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-30_15:34:36 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T17:12:29.617 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa; will attempt to use it 2023-10-30T17:12:29.618 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks 2023-10-30T17:12:29.619 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T17:12:29.620 INFO:teuthology.task.internal:Checking packages... 2023-10-30T17:12:29.644 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'fee4af45ddecf08e014b4503e218408d9e5c5be2' 2023-10-30T17:12:29.644 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T17:12:29.644 INFO:teuthology.packaging:ref: None 2023-10-30T17:12:29.645 INFO:teuthology.packaging:tag: None 2023-10-30T17:12:29.645 INFO:teuthology.packaging:branch: wip-yuri10-testing-2023-10-27-0804 2023-10-30T17:12:29.645 INFO:teuthology.packaging:sha1: fee4af45ddecf08e014b4503e218408d9e5c5be2 2023-10-30T17:12:29.646 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri10-testing-2023-10-27-0804 2023-10-30T17:12:29.871 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6865.gfee4af45 2023-10-30T17:12:29.872 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T17:12:29.879 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T17:12:29.879 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T17:12:29.913 INFO:teuthology.task.internal:Saving configuration 2023-10-30T17:12:29.929 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T17:12:29.947 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T17:12:29.972 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 17:07:24.953494', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEsA2T3Z7BhEScoWd2R+zeQ3a4mkh6Qpyg5hRdYQ+wKe/dOBtKMtrJRAdobB9gMKKCdoByietgpKaqqfGE15ufg='} 2023-10-30T17:12:29.999 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 17:07:24.951707', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKldg/iMffn8uQIfGj3LsSXEf/wPuI0S90sdsB603K8/X9pi5GwOwl1hilfS5DV06t4Vp5SxbwhF17bGKJ+zSg='} 2023-10-30T17:12:30.000 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T17:12:30.006 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-10-30T17:12:30.006 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-10-30T17:12:30.007 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T17:12:30.100 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4c2ce0ba60>, signals=[15]) 2023-10-30T17:12:30.101 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T17:12:30.109 INFO:teuthology.task.internal:Opening connections... 2023-10-30T17:12:30.109 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T17:12:30.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T17:12:30.198 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T17:12:30.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T17:12:30.282 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T17:12:30.289 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2023-10-30T17:12:30.307 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2023-10-30T17:12:30.308 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2023-10-30T17:12:30.367 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2023-10-30T17:12:30.367 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2023-10-30T17:12:30.368 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2023-10-30T17:12:30.368 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2023-10-30T17:12:30.368 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2023-10-30T17:12:30.368 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2023-10-30T17:12:30.369 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-30T17:12:30.369 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2023-10-30T17:12:30.369 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2023-10-30T17:12:30.369 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-30T17:12:30.370 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2023-10-30T17:12:30.370 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-30T17:12:30.370 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-30T17:12:30.370 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-30T17:12:30.371 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2023-10-30T17:12:30.400 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2023-10-30T17:12:30.417 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2023-10-30T17:12:30.418 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2023-10-30T17:12:30.473 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2023-10-30T17:12:30.474 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2023-10-30T17:12:30.474 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2023-10-30T17:12:30.474 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2023-10-30T17:12:30.475 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2023-10-30T17:12:30.475 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2023-10-30T17:12:30.475 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-30T17:12:30.475 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2023-10-30T17:12:30.475 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2023-10-30T17:12:30.476 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-30T17:12:30.476 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2023-10-30T17:12:30.476 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-30T17:12:30.476 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-30T17:12:30.476 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-30T17:12:30.477 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2023-10-30T17:12:30.500 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T17:12:30.509 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T17:12:30.530 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T17:12:30.531 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2023-10-30T17:12:30.533 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2023-10-30T17:12:30.549 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T17:12:30.555 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T17:12:30.555 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2023-10-30T17:12:30.594 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2023-10-30T17:12:30.666 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T17:12:30.725 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T17:12:30.740 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-30T17:12:30.741 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-30T17:12:30.741 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T17:12:30.742 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T17:12:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T17:12:30.759 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2023-10-30T17:12:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T17:12:30.760 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-10-30T17:12:30.816 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-361.el9.x86_64 2023-10-30T17:12:30.816 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-361.el9.x86_64 2023-10-30T17:12:30.817 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2023-10-30T17:12:30.820 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-361.el9.x86_64 2023-10-30T17:12:30.821 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-361.el9.x86_64 2023-10-30T17:12:30.821 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2023-10-30T17:12:32.820 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 17 MB/s | 6.3 MB 00:00 2023-10-30T17:12:32.873 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-10-30T17:12:34.145 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2023-10-30T17:12:34.165 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 9.0 MB/s | 2.0 MB 00:00 2023-10-30T17:12:41.191 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 1.2 MB/s | 7.9 MB 00:06 2023-10-30T17:12:45.228 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 751 kB/s | 7.9 MB 00:10 2023-10-30T17:12:46.299 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 18 MB 00:04 2023-10-30T17:12:47.083 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 22 MB/s | 18 MB 00:00 2023-10-30T17:12:51.059 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 5.8 MB 00:00 2023-10-30T17:12:52.534 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2023-10-30T17:12:53.568 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 5.8 MB 00:03 2023-10-30T17:12:55.107 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2023-10-30T17:12:57.142 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 10 MB/s | 19 MB 00:01 2023-10-30T17:13:01.700 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-10-30T17:13:03.651 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 1.8 MB/s | 19 MB 00:11 2023-10-30T17:13:03.773 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-30T17:13:03.774 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-30T17:13:03.858 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:13:03.866 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:13:03.866 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-10-30T17:13:03.866 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:13:03.866 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:13:03.867 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-30T17:13:03.867 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-30T17:13:03.867 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-30T17:13:03.867 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-30T17:13:03.867 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-30T17:13:03.868 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:13:03.868 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:13:03.868 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:13:03.868 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2023-10-30T17:13:03.868 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:13:03.869 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 95 M 2023-10-30T17:13:03.869 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 120 M 2023-10-30T17:13:03.869 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:13:05.260 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 4.3 MB/s | 5.3 MB 00:01 2023-10-30T17:13:05.893 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 11 MB/s | 20 MB 00:01 2023-10-30T17:13:06.877 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 13 MB/s | 38 MB 00:02 2023-10-30T17:13:07.202 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 17 MB/s | 32 MB 00:01 2023-10-30T17:13:07.202 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:13:07.203 INFO:teuthology.orchestra.run.smithi170.stdout:Total 28 MB/s | 95 MB 00:03 2023-10-30T17:13:08.043 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:13:08.392 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 5.6 kB/s | 1.7 kB 00:00 2023-10-30T17:13:08.418 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:13:08.418 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:13:09.739 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:13:09.740 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:13:10.432 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-30T17:13:10.433 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-30T17:13:10.519 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:13:10.531 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:13:10.531 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2023-10-30T17:13:10.532 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:13:10.532 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:13:10.532 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-30T17:13:10.532 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2023-10-30T17:13:10.532 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-30T17:13:10.533 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-30T17:13:10.533 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-30T17:13:10.533 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:13:10.533 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:13:10.533 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:13:10.534 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2023-10-30T17:13:10.534 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:13:10.535 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 95 M 2023-10-30T17:13:10.535 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 120 M 2023-10-30T17:13:10.535 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:13:12.177 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 3.9 MB/s | 5.3 MB 00:01 2023-10-30T17:13:12.654 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:13:12.810 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 10 MB/s | 20 MB 00:02 2023-10-30T17:13:13.281 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-30T17:13:13.419 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T17:13:13.711 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 13 MB/s | 38 MB 00:02 2023-10-30T17:13:13.986 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 18 MB/s | 32 MB 00:01 2023-10-30T17:13:13.986 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:13:13.987 INFO:teuthology.orchestra.run.smithi177.stdout:Total 27 MB/s | 95 MB 00:03 2023-10-30T17:13:14.836 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:13:15.204 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:13:15.205 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:13:16.181 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T17:13:16.382 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T17:13:16.520 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:13:16.520 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:13:19.133 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:13:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-30T17:13:19.895 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T17:13:20.952 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T17:13:21.154 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:13:22.653 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T17:13:22.839 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T17:13:25.764 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:13:27.463 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T17:13:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:13:32.215 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:13:54.552 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:13:54.593 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:13:55.036 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:13:55.036 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-30T17:13:55.036 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T17:13:55.037 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T17:13:55.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:13:55.442 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:13:55.443 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:13:55.443 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-30T17:13:55.443 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-30T17:13:55.443 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-30T17:13:55.443 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-30T17:13:55.444 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:13:55.444 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:13:55.953 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T17:13:56.103 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T17:13:56.104 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T17:13:56.104 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-30T17:13:56.105 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-30T17:13:56.105 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-30T17:13:56.105 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-30T17:13:56.105 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2023-10-30T17:13:56.106 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T17:13:56.223 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T17:13:56.224 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T17:13:56.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2023-10-30T17:13:56.265 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2023-10-30T17:13:56.266 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2023-10-30T17:13:56.266 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2023-10-30T17:13:56.266 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 68.el9 2023-10-30T17:13:56.266 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2023-10-30T17:13:56.267 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-30T17:13:56.267 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2023-10-30T17:13:56.267 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8264254 2023-10-30T17:13:56.267 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2023-10-30T17:13:56.267 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-30T17:13:56.268 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-30T17:13:56.268 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-30T17:13:56.268 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-30T17:13:56.268 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2023-10-30T17:13:56.269 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2023-10-30T17:13:56.269 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-30T17:13:56.269 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2023-10-30T17:13:56.269 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2023-10-30T17:13:56.269 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:13:56.270 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-30T17:13:56.270 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-30T17:13:56.270 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-30T17:13:56.270 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2023-10-30T17:13:56.270 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:13:56.271 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2023-10-30T17:13:56.271 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-30T17:13:56.272 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 5.14.0-378.el9.x86_64 2023-10-30T17:13:56.272 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-30T17:13:56.961 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2023-10-30T17:13:58.335 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-30T17:13:58.372 INFO:teuthology.orchestra.run.smithi170.stderr:done 2023-10-30T17:13:58.374 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-10-30T17:13:58.390 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.IKoz6cbug0 2023-10-30T17:13:58.391 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IKoz6cbug0 2023-10-30T17:13:58.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.IKoz6cbug0 2023-10-30T17:13:58.593 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.IKoz6cbug0 is 6KB 2023-10-30T17:13:58.642 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.IKoz6cbug0 2023-10-30T17:13:58.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2023-10-30T17:13:58.721 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-30T17:13:58.721 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-30T17:13:58.722 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-30T17:13:58.722 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-30T17:13:58.723 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-30T17:13:58.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2023-10-30T17:14:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:14:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:14:01.470 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:14:01.470 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-30T17:14:01.471 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T17:14:01.471 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T17:14:01.881 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T17:14:01.881 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:14:01.881 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:14:01.882 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-30T17:14:01.882 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-30T17:14:01.882 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-30T17:14:01.882 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-30T17:14:01.882 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:14:01.885 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:14:02.381 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T17:14:02.536 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T17:14:02.536 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T17:14:02.536 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-30T17:14:02.537 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-30T17:14:02.537 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-30T17:14:02.537 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-30T17:14:02.537 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2023-10-30T17:14:02.538 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T17:14:02.654 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T17:14:02.654 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T17:14:02.654 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2023-10-30T17:14:02.688 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2023-10-30T17:14:02.689 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2023-10-30T17:14:02.689 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2023-10-30T17:14:02.689 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 68.el9 2023-10-30T17:14:02.690 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2023-10-30T17:14:02.690 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-30T17:14:02.690 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2023-10-30T17:14:02.690 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8264254 2023-10-30T17:14:02.691 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2023-10-30T17:14:02.691 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-30T17:14:02.691 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-30T17:14:02.691 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-30T17:14:02.691 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-30T17:14:02.692 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2023-10-30T17:14:02.692 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2023-10-30T17:14:02.692 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-30T17:14:02.692 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2023-10-30T17:14:02.692 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2023-10-30T17:14:02.693 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:14:02.693 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-30T17:14:02.693 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-30T17:14:02.693 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-30T17:14:02.693 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2023-10-30T17:14:02.694 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:14:02.694 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2023-10-30T17:14:02.695 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-30T17:14:02.695 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 5.14.0-378.el9.x86_64 2023-10-30T17:14:02.696 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-30T17:14:03.312 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2023-10-30T17:14:04.839 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-30T17:14:04.857 INFO:teuthology.orchestra.run.smithi177.stderr:done 2023-10-30T17:14:04.868 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2023-10-30T17:14:04.923 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.wswBs8yK6B 2023-10-30T17:14:04.923 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wswBs8yK6B 2023-10-30T17:14:04.991 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.wswBs8yK6B 2023-10-30T17:14:05.129 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.wswBs8yK6B is 6KB 2023-10-30T17:14:05.178 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.wswBs8yK6B 2023-10-30T17:14:05.192 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2023-10-30T17:14:05.257 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-30T17:14:05.257 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-30T17:14:05.258 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-30T17:14:05.258 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-30T17:14:05.259 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-30T17:14:05.405 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2023-10-30T17:14:28.914 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-30T17:14:28.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-30T17:14:28.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T17:14:35.415 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-30T17:14:35.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-10-30T17:14:35.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T17:15:10.079 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2023-10-30T17:15:19.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-10-30T17:15:19.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T17:15:28.926 DEBUG:teuthology.orchestra.remote:timed out 2023-10-30T17:15:37.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2023-10-30T17:15:37.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-30T17:15:37.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T17:15:38.272 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-10-30T17:15:38.753 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-30T17:15:38.754 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-10-30T17:15:38.754 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-10-30T17:15:38.767 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-378.el9.x86_64 2023-10-30T17:15:38.768 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-30T17:15:38.768 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T17:15:38.768 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-30T17:15:39.770 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-30T17:15:39.770 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-30T17:15:39.863 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2023-10-30T17:15:39.902 DEBUG:teuthology.parallel:result is None 2023-10-30T17:15:49.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-10-30T17:15:49.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T17:15:49.788 DEBUG:teuthology.orchestra.run.smithi177:> true 2023-10-30T17:15:50.161 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-10-30T17:15:50.161 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-378.el9.x86_64"... 2023-10-30T17:15:50.162 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2023-10-30T17:15:50.176 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-378.el9.x86_64 2023-10-30T17:15:50.176 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-30T17:15:50.177 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T17:15:50.177 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-30T17:15:51.178 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-30T17:15:51.179 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-30T17:15:51.259 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2023-10-30T17:15:51.304 DEBUG:teuthology.parallel:result is None 2023-10-30T17:15:51.305 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T17:15:51.313 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T17:15:51.313 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T17:15:51.317 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T17:15:51.333 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T17:15:51.357 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T17:15:51.409 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T17:15:51.409 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T17:15:51.412 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T17:15:51.541 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T17:15:51.548 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T17:15:51.549 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T17:15:51.551 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T17:15:51.583 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T17:15:51.591 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T17:15:51.597 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T17:15:51.604 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T17:15:51.606 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T17:15:51.613 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T17:15:51.614 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T17:15:51.642 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T17:15:51.670 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T17:15:51.679 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T17:15:51.680 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T17:15:51.713 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T17:15:51.727 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T17:15:51.812 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T17:15:51.907 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T17:15:51.951 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T17:15:51.993 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:15:51.993 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T17:15:52.061 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T17:15:52.103 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T17:15:52.152 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T17:15:52.230 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T17:15:52.269 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:15:52.269 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T17:15:52.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2023-10-30T17:15:52.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2023-10-30T17:15:52.391 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T17:15:52.429 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T17:15:52.830 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T17:15:52.838 INFO:teuthology.task.internal:Starting timer... 2023-10-30T17:15:52.838 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T17:15:52.888 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T17:15:52.936 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2023-10-30T17:15:52.987 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2023-10-30T17:15:52.989 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2023-10-30T17:15:53.135 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2023-10-30T17:15:53.136 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T17:15:53.137 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2023-10-30T17:15:53.171 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2023-10-30T17:15:53.172 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2023-10-30T17:15:53.214 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2023-10-30T17:15:53.214 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi170.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2023-10-30T17:15:53.215 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T17:15:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T17:15:53.244 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T17:15:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T17:15:53.300 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T17:15:53.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2023-10-30T17:15:53.331 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2023-10-30T17:15:53.566 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T17:15:53.579 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-30T17:15:53.580 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T17:15:53.597 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T17:15:53.669 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi170.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T17:21:23.467 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi170.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2023-10-30T17:21:23.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-30T17:21:23.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T17:21:23.553 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-10-30T17:21:23.621 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-30T17:21:23.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-10-30T17:21:23.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T17:21:23.704 DEBUG:teuthology.orchestra.run.smithi177:> true 2023-10-30T17:21:23.773 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-10-30T17:21:23.773 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T17:21:23.787 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T17:21:23.788 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T17:21:23.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T17:21:23.791 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T17:21:23.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T17:21:23.819 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-30T17:21:23.836 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-30T17:21:23.843 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-30T17:21:23.860 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-30T17:21:23.864 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2023-10-30T17:21:23.878 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2023-10-30T17:21:23.889 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2023-10-30T17:21:23.894 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-30T17:21:23.902 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2023-10-30T17:21:23.910 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-30T17:21:23.919 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-30T17:21:23.935 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-30T17:21:23.953 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2023-10-30T17:21:23.956 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T17:21:23.956 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-10-30T17:21:23.956 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T17:21:23.956 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T17:21:23.957 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T17:21:23.957 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T17:21:23.978 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2023-10-30T17:21:23.981 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T17:21:23.981 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2023-10-30T17:21:23.981 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T17:21:23.981 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T17:21:23.982 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T17:21:23.982 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T17:21:23.983 INFO:teuthology.run_tasks:Running task install... 2023-10-30T17:21:23.991 DEBUG:teuthology.task.install:project ceph 2023-10-30T17:21:23.992 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'fee4af45ddecf08e014b4503e218408d9e5c5be2'}} 2023-10-30T17:21:23.992 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'fee4af45ddecf08e014b4503e218408d9e5c5be2'} 2023-10-30T17:21:23.992 INFO:teuthology.task.install:Using flavor: default 2023-10-30T17:21:23.998 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-10-30T17:21:23.998 INFO:teuthology.task.install:extra packages: [] 2023-10-30T17:21:23.999 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'fee4af45ddecf08e014b4503e218408d9e5c5be2', 'tag': None, 'wait_for_package': False} 2023-10-30T17:21:24.000 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fee4af45ddecf08e014b4503e218408d9e5c5be2 2023-10-30T17:21:24.002 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'fee4af45ddecf08e014b4503e218408d9e5c5be2', 'tag': None, 'wait_for_package': False} 2023-10-30T17:21:24.002 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fee4af45ddecf08e014b4503e218408d9e5c5be2 2023-10-30T17:21:24.150 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-27-0804/fee4af45ddecf08e014b4503e218408d9e5c5be2/centos/9/flavors/default/ 2023-10-30T17:21:24.150 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6865.gfee4af45 2023-10-30T17:21:24.157 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-27-0804/fee4af45ddecf08e014b4503e218408d9e5c5be2/centos/9/flavors/default/ 2023-10-30T17:21:24.157 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6865.gfee4af45 2023-10-30T17:21:24.273 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-27-0804/fee4af45ddecf08e014b4503e218408d9e5c5be2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-27-0804/fee4af45ddecf08e014b4503e218408d9e5c5be2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-27-0804/fee4af45ddecf08e014b4503e218408d9e5c5be2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-30T17:21:24.273 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:21:24.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-30T17:21:24.290 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-27-0804/fee4af45ddecf08e014b4503e218408d9e5c5be2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-27-0804/fee4af45ddecf08e014b4503e218408d9e5c5be2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-27-0804/fee4af45ddecf08e014b4503e218408d9e5c5be2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-30T17:21:24.290 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:21:24.290 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-30T17:21:24.302 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-10-30T17:21:24.302 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/fee4af45ddecf08e014b4503e218408d9e5c5be2/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-30T17:21:24.318 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-10-30T17:21:24.319 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/fee4af45ddecf08e014b4503e218408d9e5c5be2/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-30T17:21:24.370 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-30T17:21:24.385 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-30T17:21:24.449 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-30T17:21:24.464 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-30T17:21:24.515 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2023-10-30T17:21:24.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2023-10-30T17:21:24.530 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2023-10-30T17:21:24.532 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2023-10-30T17:21:24.755 INFO:teuthology.orchestra.run.smithi170.stdout:72 files removed 2023-10-30T17:21:24.774 INFO:teuthology.orchestra.run.smithi177.stdout:72 files removed 2023-10-30T17:21:24.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw 2023-10-30T17:21:24.801 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw 2023-10-30T17:21:25.454 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 233 kB/s | 78 kB 00:00 2023-10-30T17:21:25.471 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 210 kB/s | 78 kB 00:00 2023-10-30T17:21:25.788 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 33 kB/s | 9.4 kB 00:00 2023-10-30T17:21:25.839 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 30 kB/s | 9.4 kB 00:00 2023-10-30T17:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 6.9 kB/s | 1.9 kB 00:00 2023-10-30T17:21:26.164 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 6.5 kB/s | 1.9 kB 00:00 2023-10-30T17:21:26.508 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 17 MB/s | 6.3 MB 00:00 2023-10-30T17:21:26.542 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 19 MB/s | 6.3 MB 00:00 2023-10-30T17:21:27.800 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 8.0 MB/s | 2.0 MB 00:00 2023-10-30T17:21:27.825 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 8.1 MB/s | 2.0 MB 00:00 2023-10-30T17:21:28.615 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 7.9 MB 00:00 2023-10-30T17:21:30.920 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 7.9 MB 00:02 2023-10-30T17:21:34.397 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 18 MB 00:02 2023-10-30T17:21:35.104 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 3.3 MB/s | 18 MB 00:05 2023-10-30T17:21:39.763 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 5.8 MB 00:02 2023-10-30T17:21:41.165 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 5.8 MB 00:02 2023-10-30T17:21:41.179 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 15 kB 00:00 2023-10-30T17:21:42.022 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 27 MB/s | 19 MB 00:00 2023-10-30T17:21:42.634 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2023-10-30T17:21:46.456 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2023-10-30T17:21:48.205 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 19 MB 00:05 2023-10-30T17:21:48.832 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:21:48.839 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:21:48.839 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-30T17:21:48.839 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:21:48.840 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:21:48.840 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 8.6 M 2023-10-30T17:21:48.840 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2023-10-30T17:21:48.840 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 3.3 M 2023-10-30T17:21:48.840 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 3.0 M 2023-10-30T17:21:48.840 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-30T17:21:48.841 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options 2023-10-30T17:21:48.841 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-30T17:21:48.841 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 5.2 M 2023-10-30T17:21:48.841 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 19 M 2023-10-30T17:21:48.841 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 24 k 2023-10-30T17:21:48.842 INFO:teuthology.orchestra.run.smithi170.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-30T17:21:48.842 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-30T17:21:48.842 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-30T17:21:48.842 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-30T17:21:48.843 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 716 k 2023-10-30T17:21:48.843 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-30T17:21:48.843 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-30T17:21:48.843 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-30T17:21:48.843 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 473 k 2023-10-30T17:21:48.843 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-30T17:21:48.844 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 4.8 M 2023-10-30T17:21:48.844 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-30T17:21:48.844 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-30T17:21:48.844 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-30T17:21:48.844 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse 2023-10-30T17:21:48.845 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 45 k 2023-10-30T17:21:48.845 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 127 k 2023-10-30T17:21:48.845 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 161 k 2023-10-30T17:21:48.845 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-30T17:21:48.845 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 321 k 2023-10-30T17:21:48.846 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 298 k 2023-10-30T17:21:48.846 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 99 k 2023-10-30T17:21:48.846 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-30T17:21:48.846 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-30T17:21:48.846 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:21:48.847 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:21:48.847 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:21:48.847 INFO:teuthology.orchestra.run.smithi170.stdout:Install 28 Packages 2023-10-30T17:21:48.847 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 2 Packages 2023-10-30T17:21:48.847 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:21:48.848 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 55 M 2023-10-30T17:21:48.848 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:21:50.315 INFO:teuthology.orchestra.run.smithi170.stdout:(1/30): ceph-base-18.0.0-6865.gfee4af45.el9.x86 4.5 MB/s | 5.2 MB 00:01 2023-10-30T17:21:50.382 INFO:teuthology.orchestra.run.smithi170.stdout:(2/30): ceph-selinux-18.0.0-6865.gfee4af45.el9. 363 kB/s | 24 kB 00:00 2023-10-30T17:21:50.549 INFO:teuthology.orchestra.run.smithi170.stdout:(3/30): libcephfs2-18.0.0-6865.gfee4af45.el9.x8 4.2 MB/s | 716 kB 00:00 2023-10-30T17:21:50.666 INFO:teuthology.orchestra.run.smithi170.stdout:(4/30): libradosstriper1-18.0.0-6865.gfee4af45. 4.0 MB/s | 473 kB 00:00 2023-10-30T17:21:50.849 INFO:teuthology.orchestra.run.smithi170.stdout:(5/30): ceph-radosgw-18.0.0-6865.gfee4af45.el9. 5.1 MB/s | 8.6 MB 00:01 2023-10-30T17:21:50.933 INFO:teuthology.orchestra.run.smithi170.stdout:(6/30): python3-ceph-argparse-18.0.0-6865.gfee4 537 kB/s | 45 kB 00:00 2023-10-30T17:21:51.025 INFO:teuthology.orchestra.run.smithi170.stdout:(7/30): python3-ceph-common-18.0.0-6865.gfee4af 1.4 MB/s | 127 kB 00:00 2023-10-30T17:21:51.125 INFO:teuthology.orchestra.run.smithi170.stdout:(8/30): python3-cephfs-18.0.0-6865.gfee4af45.el 1.6 MB/s | 161 kB 00:00 2023-10-30T17:21:51.234 INFO:teuthology.orchestra.run.smithi170.stdout:(9/30): python3-rados-18.0.0-6865.gfee4af45.el9 2.9 MB/s | 321 kB 00:00 2023-10-30T17:21:51.376 INFO:teuthology.orchestra.run.smithi170.stdout:(10/30): librgw2-18.0.0-6865.gfee4af45.el9.x86_ 6.7 MB/s | 4.8 MB 00:00 2023-10-30T17:21:51.402 INFO:teuthology.orchestra.run.smithi170.stdout:(11/30): python3-rbd-18.0.0-6865.gfee4af45.el9. 1.7 MB/s | 298 kB 00:00 2023-10-30T17:21:51.426 INFO:teuthology.orchestra.run.smithi170.stdout:(12/30): python3-rgw-18.0.0-6865.gfee4af45.el9. 1.9 MB/s | 99 kB 00:00 2023-10-30T17:21:51.635 INFO:teuthology.orchestra.run.smithi170.stdout:(13/30): ceph-common-18.0.0-6865.gfee4af45.el9. 7.7 MB/s | 19 MB 00:02 2023-10-30T17:21:51.693 INFO:teuthology.orchestra.run.smithi170.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 362 kB/s | 106 kB 00:00 2023-10-30T17:21:51.727 INFO:teuthology.orchestra.run.smithi170.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.9 MB/s | 661 kB 00:00 2023-10-30T17:21:51.752 INFO:teuthology.orchestra.run.smithi170.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 282 kB 00:00 2023-10-30T17:21:51.777 INFO:teuthology.orchestra.run.smithi170.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2023-10-30T17:21:51.803 INFO:teuthology.orchestra.run.smithi170.stdout:(18/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 452 kB/s | 159 kB 00:00 2023-10-30T17:21:51.853 INFO:teuthology.orchestra.run.smithi170.stdout:(19/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2023-10-30T17:21:52.112 INFO:teuthology.orchestra.run.smithi170.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 332 kB/s | 111 kB 00:00 2023-10-30T17:21:52.195 INFO:teuthology.orchestra.run.smithi170.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 787 kB/s | 309 kB 00:00 2023-10-30T17:21:52.229 INFO:teuthology.orchestra.run.smithi170.stdout:(22/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 208 kB/s | 24 kB 00:00 2023-10-30T17:21:52.287 INFO:teuthology.orchestra.run.smithi170.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 528 kB/s | 49 kB 00:00 2023-10-30T17:21:52.313 INFO:teuthology.orchestra.run.smithi170.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 806 kB/s | 67 kB 00:00 2023-10-30T17:21:52.538 INFO:teuthology.orchestra.run.smithi170.stdout:(25/30): libarrow-9.0.0-9.el9.x86_64.rpm 6.5 MB/s | 4.4 MB 00:00 2023-10-30T17:21:52.580 INFO:teuthology.orchestra.run.smithi170.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 718 kB/s | 192 kB 00:00 2023-10-30T17:21:52.622 INFO:teuthology.orchestra.run.smithi170.stdout:(27/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.4 MB/s | 837 kB 00:00 2023-10-30T17:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-10-30T17:21:52.785 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2023-10-30T17:21:52.931 INFO:teuthology.orchestra.run.smithi170.stdout:(29/30): librbd1-18.0.0-6865.gfee4af45.el9.x86_ 9.8 MB/s | 3.0 MB 00:00 2023-10-30T17:21:53.022 INFO:teuthology.orchestra.run.smithi170.stdout:(30/30): librados2-18.0.0-6865.gfee4af45.el9.x8 7.5 MB/s | 3.3 MB 00:00 2023-10-30T17:21:53.027 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:21:53.028 INFO:teuthology.orchestra.run.smithi170.stdout:Total 13 MB/s | 55 MB 00:04 2023-10-30T17:21:53.040 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:21:53.077 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:21:53.078 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:21:53.563 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:21:53.564 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:21:54.969 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:21:55.035 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-30T17:21:55.089 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-30T17:21:55.144 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-30T17:21:55.152 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:21:55.162 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:21:55.163 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2023-10-30T17:21:55.163 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:21:55.163 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:21:55.163 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 8.6 M 2023-10-30T17:21:55.163 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2023-10-30T17:21:55.164 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 3.3 M 2023-10-30T17:21:55.164 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 3.0 M 2023-10-30T17:21:55.164 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2023-10-30T17:21:55.164 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options 2023-10-30T17:21:55.164 INFO:teuthology.orchestra.run.smithi177.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-30T17:21:55.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 5.2 M 2023-10-30T17:21:55.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 19 M 2023-10-30T17:21:55.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 24 k 2023-10-30T17:21:55.165 INFO:teuthology.orchestra.run.smithi177.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-30T17:21:55.166 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-30T17:21:55.166 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-30T17:21:55.166 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-30T17:21:55.166 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 716 k 2023-10-30T17:21:55.166 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-30T17:21:55.167 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-30T17:21:55.167 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-30T17:21:55.167 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 473 k 2023-10-30T17:21:55.167 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-30T17:21:55.167 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 4.8 M 2023-10-30T17:21:55.167 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-30T17:21:55.168 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-30T17:21:55.168 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-30T17:21:55.168 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse 2023-10-30T17:21:55.168 INFO:teuthology.orchestra.run.smithi177.stdout: x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 45 k 2023-10-30T17:21:55.168 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 127 k 2023-10-30T17:21:55.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 161 k 2023-10-30T17:21:55.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-30T17:21:55.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 321 k 2023-10-30T17:21:55.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 298 k 2023-10-30T17:21:55.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 99 k 2023-10-30T17:21:55.170 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-30T17:21:55.170 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-30T17:21:55.170 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:21:55.170 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:21:55.170 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:21:55.171 INFO:teuthology.orchestra.run.smithi177.stdout:Install 28 Packages 2023-10-30T17:21:55.171 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 2 Packages 2023-10-30T17:21:55.171 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:21:55.172 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 55 M 2023-10-30T17:21:55.172 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:21:55.535 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-30T17:21:55.568 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:18.0.0-6865.gfee4af45.el9.x86_64 5/32 2023-10-30T17:21:55.659 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:18.0.0-6865.gfee4af45.el9.x86_64 5/32 2023-10-30T17:21:55.741 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:18.0.0-6865.gfee4af45.el9.x86_64 6/32 2023-10-30T17:21:55.783 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:18.0.0-6865.gfee4af45.el9.x86_64 7/32 2023-10-30T17:21:55.853 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:18.0.0-6865.gfee4af45.el9.x86_64 7/32 2023-10-30T17:21:55.882 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-30T17:21:55.926 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:18.0.0-6865.gfee4af45.el9. 9/32 2023-10-30T17:21:55.991 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:18.0.0-6865.gfee4af45.el9.x86_64 10/32 2023-10-30T17:21:56.011 INFO:teuthology.orchestra.run.smithi177.stdout:(1/30): ceph-base-18.0.0-6865.gfee4af45.el9.x86 7.2 MB/s | 5.2 MB 00:00 2023-10-30T17:21:56.018 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:18.0.0-6865.gfee4af45.el9.x86_6 11/32 2023-10-30T17:21:56.078 INFO:teuthology.orchestra.run.smithi177.stdout:(2/30): ceph-selinux-18.0.0-6865.gfee4af45.el9. 363 kB/s | 24 kB 00:00 2023-10-30T17:21:56.084 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6865.gfee4af45.el9.x86_6 11/32 2023-10-30T17:21:56.186 INFO:teuthology.orchestra.run.smithi177.stdout:(3/30): libcephfs2-18.0.0-6865.gfee4af45.el9.x8 6.5 MB/s | 716 kB 00:00 2023-10-30T17:21:56.190 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-30T17:21:56.218 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-30T17:21:56.258 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-30T17:21:56.278 INFO:teuthology.orchestra.run.smithi177.stdout:(4/30): libradosstriper1-18.0.0-6865.gfee4af45. 5.1 MB/s | 473 kB 00:00 2023-10-30T17:21:56.290 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-30T17:21:56.317 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-30T17:21:56.345 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-30T17:21:56.378 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T17:21:56.679 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-30T17:21:56.708 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:18.0.0-6865.gfee4af45.el9.x86_64 20/32 2023-10-30T17:21:56.797 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:18.0.0-6865.gfee4af45.el9.x86_64 20/32 2023-10-30T17:21:56.829 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:18.0.0-6865.gfee4af45.el9.x86_64 21/32 2023-10-30T17:21:56.837 INFO:teuthology.orchestra.run.smithi177.stdout:(5/30): librgw2-18.0.0-6865.gfee4af45.el9.x86_6 8.6 MB/s | 4.8 MB 00:00 2023-10-30T17:21:56.865 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-30T17:21:57.029 INFO:teuthology.orchestra.run.smithi177.stdout:(6/30): ceph-common-18.0.0-6865.gfee4af45.el9.x 11 MB/s | 19 MB 00:01 2023-10-30T17:21:57.054 INFO:teuthology.orchestra.run.smithi177.stdout:(7/30): python3-ceph-argparse-18.0.0-6865.gfee4 208 kB/s | 45 kB 00:00 2023-10-30T17:21:57.079 INFO:teuthology.orchestra.run.smithi177.stdout:(8/30): python3-ceph-common-18.0.0-6865.gfee4af 2.5 MB/s | 127 kB 00:00 2023-10-30T17:21:57.104 INFO:teuthology.orchestra.run.smithi177.stdout:(9/30): python3-cephfs-18.0.0-6865.gfee4af45.el 3.1 MB/s | 161 kB 00:00 2023-10-30T17:21:57.130 INFO:teuthology.orchestra.run.smithi177.stdout:(10/30): python3-rados-18.0.0-6865.gfee4af45.el 6.2 MB/s | 321 kB 00:00 2023-10-30T17:21:57.163 INFO:teuthology.orchestra.run.smithi177.stdout:(11/30): python3-rbd-18.0.0-6865.gfee4af45.el9. 5.0 MB/s | 298 kB 00:00 2023-10-30T17:21:57.180 INFO:teuthology.orchestra.run.smithi177.stdout:(12/30): python3-rgw-18.0.0-6865.gfee4af45.el9. 1.9 MB/s | 99 kB 00:00 2023-10-30T17:21:57.280 INFO:teuthology.orchestra.run.smithi177.stdout:(13/30): ceph-radosgw-18.0.0-6865.gfee4af45.el9 4.3 MB/s | 8.6 MB 00:01 2023-10-30T17:21:57.290 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-30T17:21:57.319 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:18.0.0-6865.gfee4af45.el9.x86_64 24/32 2023-10-30T17:21:57.390 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:18.0.0-6865.gfee4af45.el9.x86_64 24/32 2023-10-30T17:21:57.397 INFO:teuthology.orchestra.run.smithi177.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 452 kB/s | 106 kB 00:00 2023-10-30T17:21:57.414 INFO:teuthology.orchestra.run.smithi177.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 680 kB/s | 159 kB 00:00 2023-10-30T17:21:57.448 INFO:teuthology.orchestra.run.smithi177.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2023-10-30T17:21:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:18.0.0-6865.gfee4af45.el9.x86_64 25/32 2023-10-30T17:21:57.473 INFO:teuthology.orchestra.run.smithi177.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 661 kB 00:00 2023-10-30T17:21:57.490 INFO:teuthology.orchestra.run.smithi177.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 547 kB/s | 41 kB 00:00 2023-10-30T17:21:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:18.0.0-6865.gfee4af45.el9.x8 26/32 2023-10-30T17:21:57.657 INFO:teuthology.orchestra.run.smithi177.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 664 kB/s | 111 kB 00:00 2023-10-30T17:21:57.716 INFO:teuthology.orchestra.run.smithi177.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2023-10-30T17:21:57.766 INFO:teuthology.orchestra.run.smithi177.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 488 kB/s | 24 kB 00:00 2023-10-30T17:21:57.816 INFO:teuthology.orchestra.run.smithi177.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2023-10-30T17:21:57.833 INFO:teuthology.orchestra.run.smithi177.stdout:(23/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 118 kB/s | 45 kB 00:00 2023-10-30T17:21:57.867 INFO:teuthology.orchestra.run.smithi177.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2023-10-30T17:21:57.934 INFO:teuthology.orchestra.run.smithi177.stdout:(25/30): libarrow-9.0.0-9.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2023-10-30T17:21:57.959 INFO:teuthology.orchestra.run.smithi177.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2023-10-30T17:21:58.043 INFO:teuthology.orchestra.run.smithi177.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-10-30T17:21:58.218 INFO:teuthology.orchestra.run.smithi177.stdout:(28/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.1 MB/s | 837 kB 00:00 2023-10-30T17:21:58.318 INFO:teuthology.orchestra.run.smithi177.stdout:(29/30): librados2-18.0.0-6865.gfee4af45.el9.x8 9.2 MB/s | 3.3 MB 00:00 2023-10-30T17:21:58.377 INFO:teuthology.orchestra.run.smithi177.stdout:(30/30): librbd1-18.0.0-6865.gfee4af45.el9.x86_ 9.1 MB/s | 3.0 MB 00:00 2023-10-30T17:21:58.381 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:21:58.381 INFO:teuthology.orchestra.run.smithi177.stdout:Total 17 MB/s | 55 MB 00:03 2023-10-30T17:21:58.391 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:21:58.426 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:21:58.426 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:21:58.888 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:21:58.888 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:21:59.626 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 27/32 2023-10-30T17:21:59.669 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 27/32 2023-10-30T17:22:00.223 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 27/32 2023-10-30T17:22:00.273 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:22:00.289 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:18.0.0-6865.gfee4af45.el9.x86_64 28/32 2023-10-30T17:22:00.372 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:18.0.0-6865.gfee4af45.el9.x86_64 28/32 2023-10-30T17:22:00.372 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-30T17:22:00.373 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-30T17:22:00.373 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:00.403 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-30T17:22:00.406 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:18.0.0-6865.gfee4af45.el9.x86_64 29/32 2023-10-30T17:22:00.519 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-30T17:22:00.574 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-30T17:22:00.942 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-30T17:22:00.972 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:18.0.0-6865.gfee4af45.el9.x86_64 5/32 2023-10-30T17:22:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:18.0.0-6865.gfee4af45.el9.x86_64 5/32 2023-10-30T17:22:01.157 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:18.0.0-6865.gfee4af45.el9.x86_64 6/32 2023-10-30T17:22:01.182 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:18.0.0-6865.gfee4af45.el9.x86_64 7/32 2023-10-30T17:22:01.245 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:18.0.0-6865.gfee4af45.el9.x86_64 7/32 2023-10-30T17:22:01.273 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-30T17:22:01.308 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:18.0.0-6865.gfee4af45.el9. 9/32 2023-10-30T17:22:01.366 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:18.0.0-6865.gfee4af45.el9.x86_64 10/32 2023-10-30T17:22:01.392 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:18.0.0-6865.gfee4af45.el9.x86_6 11/32 2023-10-30T17:22:01.441 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6865.gfee4af45.el9.x86_6 11/32 2023-10-30T17:22:01.546 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-30T17:22:01.575 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-30T17:22:01.615 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-30T17:22:01.647 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-30T17:22:01.675 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-30T17:22:01.710 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-30T17:22:01.743 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T17:22:02.057 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-30T17:22:02.081 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:18.0.0-6865.gfee4af45.el9.x86_64 20/32 2023-10-30T17:22:02.155 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:18.0.0-6865.gfee4af45.el9.x86_64 20/32 2023-10-30T17:22:02.196 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:18.0.0-6865.gfee4af45.el9.x86_64 21/32 2023-10-30T17:22:02.232 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-30T17:22:02.659 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-30T17:22:02.694 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:18.0.0-6865.gfee4af45.el9.x86_64 24/32 2023-10-30T17:22:02.757 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:18.0.0-6865.gfee4af45.el9.x86_64 24/32 2023-10-30T17:22:02.832 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:18.0.0-6865.gfee4af45.el9.x86_64 25/32 2023-10-30T17:22:02.862 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:18.0.0-6865.gfee4af45.el9.x8 26/32 2023-10-30T17:22:04.966 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 27/32 2023-10-30T17:22:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 27/32 2023-10-30T17:22:05.572 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 27/32 2023-10-30T17:22:05.612 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:18.0.0-6865.gfee4af45.el9.x86_64 28/32 2023-10-30T17:22:05.696 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:18.0.0-6865.gfee4af45.el9.x86_64 28/32 2023-10-30T17:22:05.697 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-30T17:22:05.697 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-30T17:22:05.697 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:05.730 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:18.0.0-6865.gfee4af45.el9.x86_64 29/32 2023-10-30T17:22:09.827 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6865.gfee4af45.el9.x86_64 29/32 2023-10-30T17:22:09.828 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2023-10-30T17:22:09.828 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2023-10-30T17:22:09.828 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2023-10-30T17:22:09.828 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2023-10-30T17:22:09.828 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2023-10-30T17:22:09.829 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2023-10-30T17:22:09.829 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2023-10-30T17:22:09.829 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:10.632 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:18.0.0-6865.gfee4af45.el9.x86_64 30/32 2023-10-30T17:22:10.656 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6865.gfee4af45.el9.x86_64 30/32 2023-10-30T17:22:10.656 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:10.656 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-30T17:22:10.656 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T17:22:10.657 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T17:22:10.657 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:10.658 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T17:22:10.736 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T17:22:10.737 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T17:22:11.834 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T17:22:11.834 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:18.0.0-6865.gfee4af45.el9.x86_64 1/32 2023-10-30T17:22:11.834 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 2/32 2023-10-30T17:22:11.835 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:18.0.0-6865.gfee4af45.el9.x86_64 3/32 2023-10-30T17:22:11.835 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:18.0.0-6865.gfee4af45.el9.x86_64 4/32 2023-10-30T17:22:11.835 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:18.0.0-6865.gfee4af45.el9.x86_64 5/32 2023-10-30T17:22:11.835 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:18.0.0-6865.gfee4af45.el9.x86_6 6/32 2023-10-30T17:22:11.835 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:18.0.0-6865.gfee4af45.el9.x86_64 7/32 2023-10-30T17:22:11.836 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6865.gfee4af45.el9. 8/32 2023-10-30T17:22:11.836 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:18.0.0-6865.gfee4af45.el9.x8 9/32 2023-10-30T17:22:11.836 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:18.0.0-6865.gfee4af45.el9.x86_64 10/32 2023-10-30T17:22:11.836 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:18.0.0-6865.gfee4af45.el9.x86_64 11/32 2023-10-30T17:22:11.837 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:18.0.0-6865.gfee4af45.el9.x86_64 12/32 2023-10-30T17:22:11.837 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:18.0.0-6865.gfee4af45.el9.x86_64 13/32 2023-10-30T17:22:11.837 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-30T17:22:11.837 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-30T17:22:11.837 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-30T17:22:11.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-30T17:22:11.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T17:22:11.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-30T17:22:11.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-30T17:22:11.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-30T17:22:11.839 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-30T17:22:11.839 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-30T17:22:11.839 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-30T17:22:11.839 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-30T17:22:11.839 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-30T17:22:11.840 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-30T17:22:11.840 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-30T17:22:11.840 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:18.0.0-6865.gfee4af45.el9.x86_64 29/32 2023-10-30T17:22:11.841 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-30T17:22:11.841 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:18.0.0-6865.gfee4af45.el9.x86_64 31/32 2023-10-30T17:22:12.901 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T17:22:12.901 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:12.902 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2023-10-30T17:22:12.902 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.902 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.902 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:22:12.902 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-30T17:22:12.903 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.903 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.903 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.903 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.903 INFO:teuthology.orchestra.run.smithi170.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-30T17:22:12.904 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-30T17:22:12.904 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-30T17:22:12.904 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-30T17:22:12.904 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.905 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-30T17:22:12.905 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-30T17:22:12.905 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-30T17:22:12.905 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.905 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-30T17:22:12.906 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.906 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-30T17:22:12.906 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-30T17:22:12.906 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-30T17:22:12.907 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.907 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.907 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.907 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-30T17:22:12.907 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.908 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.908 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:12.908 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2023-10-30T17:22:12.908 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-30T17:22:12.908 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:12.909 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:22:13.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-test 2023-10-30T17:22:13.712 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:27 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:22:13.854 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:22:13.854 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:13.855 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-30T17:22:13.855 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:13.855 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:22:13.855 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 41 M 2023-10-30T17:22:13.855 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-30T17:22:13.856 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-30T17:22:13.856 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 164 k 2023-10-30T17:22:13.856 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-30T17:22:13.856 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-30T17:22:13.856 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-30T17:22:13.857 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:13.857 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:22:13.857 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:13.857 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-10-30T17:22:13.857 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:13.859 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 42 M 2023-10-30T17:22:13.859 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 172 M 2023-10-30T17:22:13.859 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:22:14.223 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.1 MB/s | 217 kB 00:00 2023-10-30T17:22:14.306 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.7 MB/s | 300 kB 00:00 2023-10-30T17:22:14.340 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-30T17:22:14.482 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): jq-1.6-15.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2023-10-30T17:22:14.707 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): libcephsqlite-18.0.0-6865.gfee4af45.el9. 280 kB/s | 164 kB 00:00 2023-10-30T17:22:14.947 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6865.gfee4af45.el9.x86_64 29/32 2023-10-30T17:22:14.947 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2023-10-30T17:22:14.947 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2023-10-30T17:22:14.947 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2023-10-30T17:22:14.948 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2023-10-30T17:22:14.948 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2023-10-30T17:22:14.948 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2023-10-30T17:22:14.948 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2023-10-30T17:22:14.948 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:18.0.0-6865.gfee4af45.el9.x86_64 30/32 2023-10-30T17:22:15.810 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6865.gfee4af45.el9.x86_64 30/32 2023-10-30T17:22:15.811 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:15.811 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-30T17:22:15.811 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T17:22:15.812 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T17:22:15.812 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:15.812 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T17:22:15.898 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T17:22:15.898 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T17:22:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T17:22:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:18.0.0-6865.gfee4af45.el9.x86_64 1/32 2023-10-30T17:22:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 2/32 2023-10-30T17:22:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:18.0.0-6865.gfee4af45.el9.x86_64 3/32 2023-10-30T17:22:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:18.0.0-6865.gfee4af45.el9.x86_64 4/32 2023-10-30T17:22:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:18.0.0-6865.gfee4af45.el9.x86_64 5/32 2023-10-30T17:22:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:18.0.0-6865.gfee4af45.el9.x86_6 6/32 2023-10-30T17:22:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:18.0.0-6865.gfee4af45.el9.x86_64 7/32 2023-10-30T17:22:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6865.gfee4af45.el9. 8/32 2023-10-30T17:22:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:18.0.0-6865.gfee4af45.el9.x8 9/32 2023-10-30T17:22:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:18.0.0-6865.gfee4af45.el9.x86_64 10/32 2023-10-30T17:22:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:18.0.0-6865.gfee4af45.el9.x86_64 11/32 2023-10-30T17:22:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:18.0.0-6865.gfee4af45.el9.x86_64 12/32 2023-10-30T17:22:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:18.0.0-6865.gfee4af45.el9.x86_64 13/32 2023-10-30T17:22:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-30T17:22:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-30T17:22:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-30T17:22:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-30T17:22:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T17:22:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-30T17:22:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-30T17:22:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-30T17:22:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-30T17:22:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-30T17:22:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-30T17:22:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-30T17:22:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-30T17:22:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-30T17:22:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-30T17:22:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:18.0.0-6865.gfee4af45.el9.x86_64 29/32 2023-10-30T17:22:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-30T17:22:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:18.0.0-6865.gfee4af45.el9.x86_64 31/32 2023-10-30T17:22:17.924 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): ceph-test-18.0.0-6865.gfee4af45.el9.x86_ 11 MB/s | 41 MB 00:03 2023-10-30T17:22:17.927 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:17.928 INFO:teuthology.orchestra.run.smithi170.stdout:Total 10 MB/s | 42 MB 00:04 2023-10-30T17:22:17.941 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:22:17.945 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T17:22:17.945 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:17.945 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2023-10-30T17:22:17.945 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.946 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.946 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:22:17.946 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-30T17:22:17.946 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.947 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.947 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.947 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.947 INFO:teuthology.orchestra.run.smithi177.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-30T17:22:17.947 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-30T17:22:17.948 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-30T17:22:17.948 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-30T17:22:17.948 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.948 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-30T17:22:17.948 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-30T17:22:17.949 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-30T17:22:17.949 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.949 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-30T17:22:17.949 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.949 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-30T17:22:17.950 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-30T17:22:17.950 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-30T17:22:17.950 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.950 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.950 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.951 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-30T17:22:17.951 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.951 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.951 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:17.951 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2023-10-30T17:22:17.951 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-30T17:22:17.952 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:17.952 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:22:17.963 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:22:17.963 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:22:18.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test 2023-10-30T17:22:18.327 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:22:18.327 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:22:18.747 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:26 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:22:18.884 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:22:18.887 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:22:18.888 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2023-10-30T17:22:18.888 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:18.889 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:22:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 41 M 2023-10-30T17:22:18.889 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2023-10-30T17:22:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-30T17:22:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 164 k 2023-10-30T17:22:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-30T17:22:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-30T17:22:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-30T17:22:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:18.890 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:22:18.891 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:18.891 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2023-10-30T17:22:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:18.893 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 42 M 2023-10-30T17:22:18.893 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 172 M 2023-10-30T17:22:18.894 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:22:18.967 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-30T17:22:19.000 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-30T17:22:19.034 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T17:22:19.083 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-30T17:22:19.111 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:18.0.0-6865.gfee4af45.el9.x86_64 5/6 2023-10-30T17:22:19.159 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-10-30T17:22:19.209 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2023-10-30T17:22:19.234 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2023-10-30T17:22:19.327 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): libcephsqlite-18.0.0-6865.gfee4af45.el9. 521 kB/s | 164 kB 00:00 2023-10-30T17:22:19.385 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): jq-1.6-15.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-10-30T17:22:21.211 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): ceph-test-18.0.0-6865.gfee4af45.el9.x86_ 19 MB/s | 41 MB 00:02 2023-10-30T17:22:21.213 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:21.214 INFO:teuthology.orchestra.run.smithi177.stdout:Total 18 MB/s | 42 MB 00:02 2023-10-30T17:22:21.228 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:22:21.248 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:22:21.249 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:22:21.606 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:22:21.606 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:22:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:22:22.236 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-30T17:22:22.269 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-30T17:22:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T17:22:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-30T17:22:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:18.0.0-6865.gfee4af45.el9.x86_64 5/6 2023-10-30T17:22:22.951 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6865.gfee4af45.el9.x86_64 5/6 2023-10-30T17:22:23.025 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:18.0.0-6865.gfee4af45.el9.x86_64 6/6 2023-10-30T17:22:25.044 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-test-2:18.0.0-6865.gfee4af45.el9.x86_64 6/6 2023-10-30T17:22:25.044 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:18.0.0-6865.gfee4af45.el9.x86_64 1/6 2023-10-30T17:22:25.044 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:18.0.0-6865.gfee4af45.el9.x86_64 2/6 2023-10-30T17:22:25.045 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T17:22:25.045 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-30T17:22:25.045 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-30T17:22:25.476 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-30T17:22:25.476 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:25.476 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:22:25.477 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:25.477 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.6-15.el9.x86_64 2023-10-30T17:22:25.477 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:25.477 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-30T17:22:25.477 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-30T17:22:25.478 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-30T17:22:25.478 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:25.478 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:22:25.760 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph 2023-10-30T17:22:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6865.gfee4af45.el9.x86_64 5/6 2023-10-30T17:22:26.269 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:18.0.0-6865.gfee4af45.el9.x86_64 6/6 2023-10-30T17:22:26.310 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:40 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:22:26.455 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:22:26.457 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2023-10-30T17:22:26.457 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-30T17:22:26.457 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2023-10-30T17:22:26.458 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:22:26.458 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 6.5 k 2023-10-30T17:22:26.458 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-30T17:22:26.458 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 2.1 M 2023-10-30T17:22:26.459 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 1.5 M 2023-10-30T17:22:26.459 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 242 k 2023-10-30T17:22:26.459 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 4.4 M 2023-10-30T17:22:26.459 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 18 M 2023-10-30T17:22:26.459 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-30T17:22:26.459 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-30T17:22:26.460 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-30T17:22:26.460 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-30T17:22:26.460 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-30T17:22:26.460 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-30T17:22:26.460 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-30T17:22:26.461 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-30T17:22:26.461 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-30T17:22:26.461 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-30T17:22:26.461 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-30T17:22:26.461 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-30T17:22:26.462 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-30T17:22:26.462 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-30T17:22:26.462 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-30T17:22:26.462 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-30T17:22:26.462 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-30T17:22:26.463 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-30T17:22:26.463 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-30T17:22:26.463 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-30T17:22:26.463 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-30T17:22:26.463 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-30T17:22:26.464 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-30T17:22:26.464 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-30T17:22:26.464 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-30T17:22:26.464 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-30T17:22:26.464 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-30T17:22:26.464 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:26.465 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:22:26.465 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2023-10-30T17:22:26.465 INFO:teuthology.orchestra.run.smithi170.stdout:Install 33 Packages 2023-10-30T17:22:26.465 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:26.466 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 30 M 2023-10-30T17:22:26.466 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 110 M 2023-10-30T17:22:26.466 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:22:27.127 INFO:teuthology.orchestra.run.smithi170.stdout:(1/33): ceph-18.0.0-6865.gfee4af45.el9.x86_64.r 20 kB/s | 6.5 kB 00:00 2023-10-30T17:22:27.302 INFO:teuthology.orchestra.run.smithi170.stdout:(2/33): ceph-mgr-18.0.0-6865.gfee4af45.el9.x86_ 2.9 MB/s | 1.5 MB 00:00 2023-10-30T17:22:27.585 INFO:teuthology.orchestra.run.smithi170.stdout:(3/33): ceph-mds-18.0.0-6865.gfee4af45.el9.x86_ 2.7 MB/s | 2.1 MB 00:00 2023-10-30T17:22:27.686 INFO:teuthology.orchestra.run.smithi170.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6865.gfee4 2.4 MB/s | 242 kB 00:00 2023-10-30T17:22:27.744 INFO:teuthology.orchestra.run.smithi170.stdout:(5/33): ceph-mon-18.0.0-6865.gfee4af45.el9.x86_ 7.1 MB/s | 4.4 MB 00:00 2023-10-30T17:22:27.845 INFO:teuthology.orchestra.run.smithi170.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 412 kB/s | 32 kB 00:00 2023-10-30T17:22:27.870 INFO:teuthology.orchestra.run.smithi170.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 783 kB/s | 144 kB 00:00 2023-10-30T17:22:27.896 INFO:teuthology.orchestra.run.smithi170.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2023-10-30T17:22:27.921 INFO:teuthology.orchestra.run.smithi170.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-10-30T17:22:27.946 INFO:teuthology.orchestra.run.smithi170.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2023-10-30T17:22:27.971 INFO:teuthology.orchestra.run.smithi170.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-10-30T17:22:27.997 INFO:teuthology.orchestra.run.smithi170.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-10-30T17:22:28.021 INFO:teuthology.orchestra.run.smithi170.stdout:(13/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-10-30T17:22:28.379 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-2:18.0.0-6865.gfee4af45.el9.x86_64 6/6 2023-10-30T17:22:28.379 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:18.0.0-6865.gfee4af45.el9.x86_64 1/6 2023-10-30T17:22:28.379 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:18.0.0-6865.gfee4af45.el9.x86_64 2/6 2023-10-30T17:22:28.380 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T17:22:28.380 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-30T17:22:28.380 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-30T17:22:28.497 INFO:teuthology.orchestra.run.smithi170.stdout:(14/33): ceph-osd-18.0.0-6865.gfee4af45.el9.x86 15 MB/s | 18 MB 00:01 2023-10-30T17:22:28.547 INFO:teuthology.orchestra.run.smithi170.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 97 kB/s | 51 kB 00:00 2023-10-30T17:22:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-30T17:22:28.764 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:28.765 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:22:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: jq-1.6-15.el9.x86_64 2023-10-30T17:22:28.765 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-30T17:22:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-30T17:22:28.766 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-30T17:22:28.767 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:28.767 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:22:28.880 INFO:teuthology.orchestra.run.smithi170.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 121 kB/s | 43 kB 00:00 2023-10-30T17:22:28.964 INFO:teuthology.orchestra.run.smithi170.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 412 kB/s | 172 kB 00:00 2023-10-30T17:22:29.031 INFO:teuthology.orchestra.run.smithi170.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2023-10-30T17:22:29.073 INFO:teuthology.orchestra.run.smithi170.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2023-10-30T17:22:29.079 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph 2023-10-30T17:22:29.098 INFO:teuthology.orchestra.run.smithi170.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2023-10-30T17:22:29.132 INFO:teuthology.orchestra.run.smithi170.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 394 kB/s | 23 kB 00:00 2023-10-30T17:22:29.157 INFO:teuthology.orchestra.run.smithi170.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 330 kB/s | 19 kB 00:00 2023-10-30T17:22:29.183 INFO:teuthology.orchestra.run.smithi170.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2023-10-30T17:22:29.208 INFO:teuthology.orchestra.run.smithi170.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 917 kB/s | 46 kB 00:00 2023-10-30T17:22:29.242 INFO:teuthology.orchestra.run.smithi170.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2023-10-30T17:22:29.300 INFO:teuthology.orchestra.run.smithi170.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2023-10-30T17:22:29.317 INFO:teuthology.orchestra.run.smithi170.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2023-10-30T17:22:29.361 INFO:teuthology.orchestra.run.smithi170.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2023-10-30T17:22:29.376 INFO:teuthology.orchestra.run.smithi170.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2023-10-30T17:22:29.443 INFO:teuthology.orchestra.run.smithi170.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2023-10-30T17:22:29.477 INFO:teuthology.orchestra.run.smithi170.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2023-10-30T17:22:29.502 INFO:teuthology.orchestra.run.smithi170.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2023-10-30T17:22:29.625 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:37 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:22:29.677 INFO:teuthology.orchestra.run.smithi170.stdout:(33/33): python3-cryptography-36.0.1-4.el9.x86_ 747 kB/s | 1.2 MB 00:01 2023-10-30T17:22:29.684 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:29.684 INFO:teuthology.orchestra.run.smithi170.stdout:Total 9.3 MB/s | 30 MB 00:03 2023-10-30T17:22:29.718 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:22:29.758 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:22:29.758 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:22:29.769 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:22:29.773 INFO:teuthology.orchestra.run.smithi177.stdout:========================================================================================= 2023-10-30T17:22:29.774 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2023-10-30T17:22:29.774 INFO:teuthology.orchestra.run.smithi177.stdout:========================================================================================= 2023-10-30T17:22:29.774 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:22:29.774 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 6.5 k 2023-10-30T17:22:29.775 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2023-10-30T17:22:29.775 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 2.1 M 2023-10-30T17:22:29.775 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 1.5 M 2023-10-30T17:22:29.775 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 242 k 2023-10-30T17:22:29.775 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 4.4 M 2023-10-30T17:22:29.776 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 18 M 2023-10-30T17:22:29.776 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-30T17:22:29.776 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-30T17:22:29.776 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-30T17:22:29.776 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-30T17:22:29.777 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-30T17:22:29.777 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-30T17:22:29.777 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-30T17:22:29.777 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-30T17:22:29.777 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-30T17:22:29.778 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-30T17:22:29.778 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-30T17:22:29.778 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-30T17:22:29.778 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-30T17:22:29.778 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-30T17:22:29.779 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-30T17:22:29.779 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-30T17:22:29.779 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-30T17:22:29.779 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-30T17:22:29.779 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-30T17:22:29.780 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-30T17:22:29.780 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-30T17:22:29.780 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-30T17:22:29.780 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-30T17:22:29.780 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-30T17:22:29.781 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-30T17:22:29.781 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-30T17:22:29.781 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-30T17:22:29.781 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:29.781 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:22:29.782 INFO:teuthology.orchestra.run.smithi177.stdout:========================================================================================= 2023-10-30T17:22:29.782 INFO:teuthology.orchestra.run.smithi177.stdout:Install 33 Packages 2023-10-30T17:22:29.782 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:29.782 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 30 M 2023-10-30T17:22:29.782 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 110 M 2023-10-30T17:22:29.783 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:22:30.088 INFO:teuthology.orchestra.run.smithi177.stdout:(1/33): ceph-18.0.0-6865.gfee4af45.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2023-10-30T17:22:30.114 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:22:30.114 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:22:30.264 INFO:teuthology.orchestra.run.smithi177.stdout:(2/33): ceph-mgr-18.0.0-6865.gfee4af45.el9.x86_ 4.3 MB/s | 1.5 MB 00:00 2023-10-30T17:22:30.680 INFO:teuthology.orchestra.run.smithi177.stdout:(3/33): ceph-mds-18.0.0-6865.gfee4af45.el9.x86_ 2.8 MB/s | 2.1 MB 00:00 2023-10-30T17:22:30.739 INFO:teuthology.orchestra.run.smithi177.stdout:(4/33): ceph-mon-18.0.0-6865.gfee4af45.el9.x86_ 6.7 MB/s | 4.4 MB 00:00 2023-10-30T17:22:30.764 INFO:teuthology.orchestra.run.smithi177.stdout:(5/33): ceph-mgr-modules-core-18.0.0-6865.gfee4 2.8 MB/s | 242 kB 00:00 2023-10-30T17:22:30.856 INFO:teuthology.orchestra.run.smithi177.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 461 kB/s | 32 kB 00:00 2023-10-30T17:22:30.882 INFO:teuthology.orchestra.run.smithi177.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2023-10-30T17:22:30.907 INFO:teuthology.orchestra.run.smithi177.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2023-10-30T17:22:30.932 INFO:teuthology.orchestra.run.smithi177.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-10-30T17:22:30.957 INFO:teuthology.orchestra.run.smithi177.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2023-10-30T17:22:30.982 INFO:teuthology.orchestra.run.smithi177.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-10-30T17:22:31.008 INFO:teuthology.orchestra.run.smithi177.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-10-30T17:22:31.033 INFO:teuthology.orchestra.run.smithi177.stdout:(13/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-10-30T17:22:31.125 INFO:teuthology.orchestra.run.smithi177.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 556 kB/s | 51 kB 00:00 2023-10-30T17:22:31.309 INFO:teuthology.orchestra.run.smithi177.stdout:(15/33): ceph-osd-18.0.0-6865.gfee4af45.el9.x86 17 MB/s | 18 MB 00:01 2023-10-30T17:22:31.367 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:22:31.438 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-30T17:22:31.479 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-30T17:22:31.520 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-30T17:22:31.612 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-30T17:22:31.698 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-30T17:22:31.748 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-30T17:22:31.789 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-30T17:22:31.849 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-30T17:22:31.876 INFO:teuthology.orchestra.run.smithi177.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 58 kB/s | 43 kB 00:00 2023-10-30T17:22:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-30T17:22:31.917 INFO:teuthology.orchestra.run.smithi177.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 294 kB/s | 172 kB 00:00 2023-10-30T17:22:31.940 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-30T17:22:31.989 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T17:22:32.034 INFO:teuthology.orchestra.run.smithi177.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2023-10-30T17:22:32.045 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T17:22:32.143 INFO:teuthology.orchestra.run.smithi177.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2023-10-30T17:22:32.202 INFO:teuthology.orchestra.run.smithi177.stdout:(20/33): python3-cryptography-36.0.1-4.el9.x86_ 1.0 MB/s | 1.2 MB 00:01 2023-10-30T17:22:32.228 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-30T17:22:32.245 INFO:teuthology.orchestra.run.smithi177.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 974 kB/s | 358 kB 00:00 2023-10-30T17:22:32.261 INFO:teuthology.orchestra.run.smithi177.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 197 kB/s | 23 kB 00:00 2023-10-30T17:22:32.278 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-30T17:22:32.308 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-30T17:22:32.343 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-30T17:22:32.352 INFO:teuthology.orchestra.run.smithi177.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2023-10-30T17:22:32.369 INFO:teuthology.orchestra.run.smithi177.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2023-10-30T17:22:32.373 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-30T17:22:32.449 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-30T17:22:32.470 INFO:teuthology.orchestra.run.smithi177.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 675 kB/s | 79 kB 00:00 2023-10-30T17:22:32.484 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-30T17:22:32.517 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-30T17:22:32.558 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-30T17:22:32.561 INFO:teuthology.orchestra.run.smithi177.stdout:(26/33): python3-jaraco-functools-3.5.0-2.el9.n 54 kB/s | 19 kB 00:00 2023-10-30T17:22:32.585 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-30T17:22:32.595 INFO:teuthology.orchestra.run.smithi177.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2023-10-30T17:22:32.612 INFO:teuthology.orchestra.run.smithi177.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 116 kB/s | 16 kB 00:00 2023-10-30T17:22:32.673 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-30T17:22:32.696 INFO:teuthology.orchestra.run.smithi177.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2023-10-30T17:22:32.715 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-30T17:22:32.746 INFO:teuthology.orchestra.run.smithi177.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2023-10-30T17:22:32.787 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-30T17:22:32.813 INFO:teuthology.orchestra.run.smithi177.stdout:(31/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 358 kB/s | 90 kB 00:00 2023-10-30T17:22:32.863 INFO:teuthology.orchestra.run.smithi177.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2023-10-30T17:22:32.905 INFO:teuthology.orchestra.run.smithi177.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.0 MB/s | 427 kB 00:00 2023-10-30T17:22:32.912 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:32.912 INFO:teuthology.orchestra.run.smithi177.stdout:Total 9.5 MB/s | 30 MB 00:03 2023-10-30T17:22:32.923 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-30T17:22:32.945 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:22:32.984 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:22:32.985 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:22:33.041 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T17:22:33.186 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6865.gfee4af45.el9. 28/33 2023-10-30T17:22:33.212 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:18.0.0-6865.gfee4af45.el9.x86_64 29/33 2023-10-30T17:22:33.236 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6865.gfee4af45.el9.x86_64 29/33 2023-10-30T17:22:33.236 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:33.236 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-30T17:22:33.236 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T17:22:33.237 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T17:22:33.237 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:33.357 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:22:33.357 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:22:34.617 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:22:34.674 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-30T17:22:34.715 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-30T17:22:34.756 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-30T17:22:34.814 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:18.0.0-6865.gfee4af45.el9.x86_64 30/33 2023-10-30T17:22:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:18.0.0-6865.gfee4af45.el9.x86_64 30/33 2023-10-30T17:22:34.842 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:34.842 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-30T17:22:34.842 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T17:22:34.842 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T17:22:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:34.847 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-30T17:22:34.926 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-30T17:22:34.967 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-30T17:22:35.004 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-30T17:22:35.073 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-30T17:22:35.126 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-30T17:22:35.165 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-30T17:22:35.213 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T17:22:35.269 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T17:22:35.281 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:18.0.0-6865.gfee4af45.el9.x86_64 31/33 2023-10-30T17:22:35.304 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:18.0.0-6865.gfee4af45.el9.x86_64 31/33 2023-10-30T17:22:35.305 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:35.305 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-30T17:22:35.305 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T17:22:35.305 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T17:22:35.306 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:35.454 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-30T17:22:35.511 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-30T17:22:35.541 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-30T17:22:35.565 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:18.0.0-6865.gfee4af45.el9.x86_64 32/33 2023-10-30T17:22:35.576 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-30T17:22:35.588 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:18.0.0-6865.gfee4af45.el9.x86_64 32/33 2023-10-30T17:22:35.588 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:35.588 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-30T17:22:35.589 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T17:22:35.589 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T17:22:35.589 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:35.606 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-30T17:22:35.654 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:18.0.0-6865.gfee4af45.el9.x86_64 33/33 2023-10-30T17:22:35.682 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-30T17:22:35.716 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-30T17:22:35.749 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-30T17:22:35.790 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-30T17:22:35.826 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-30T17:22:35.915 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-30T17:22:35.964 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-30T17:22:36.046 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-30T17:22:36.181 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-30T17:22:36.296 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T17:22:36.451 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6865.gfee4af45.el9. 28/33 2023-10-30T17:22:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:18.0.0-6865.gfee4af45.el9.x86_64 29/33 2023-10-30T17:22:36.502 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6865.gfee4af45.el9.x86_64 29/33 2023-10-30T17:22:36.503 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:36.503 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-30T17:22:36.503 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T17:22:36.503 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T17:22:36.504 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:37.365 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-2:18.0.0-6865.gfee4af45.el9.x86_64 33/33 2023-10-30T17:22:37.366 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:18.0.0-6865.gfee4af45.el9.x86_64 1/33 2023-10-30T17:22:37.366 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:18.0.0-6865.gfee4af45.el9.x86_64 2/33 2023-10-30T17:22:37.366 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:18.0.0-6865.gfee4af45.el9.x86_64 3/33 2023-10-30T17:22:37.366 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:18.0.0-6865.gfee4af45.el9.x86_64 4/33 2023-10-30T17:22:37.366 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:18.0.0-6865.gfee4af45.el9.x86_64 5/33 2023-10-30T17:22:37.367 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6865.gfee4af45.el9. 6/33 2023-10-30T17:22:37.367 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-30T17:22:37.367 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-30T17:22:37.367 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-30T17:22:37.367 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-30T17:22:37.368 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T17:22:37.368 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T17:22:37.368 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-30T17:22:37.368 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-30T17:22:37.368 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-30T17:22:37.369 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-30T17:22:37.369 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-30T17:22:37.369 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-30T17:22:37.369 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-30T17:22:37.369 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-30T17:22:37.369 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-30T17:22:37.370 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-30T17:22:37.370 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-30T17:22:37.370 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-30T17:22:37.370 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-30T17:22:37.371 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-30T17:22:37.372 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T17:22:37.372 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-30T17:22:37.372 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-30T17:22:37.372 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-30T17:22:37.372 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-30T17:22:37.372 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-30T17:22:38.021 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:18.0.0-6865.gfee4af45.el9.x86_64 30/33 2023-10-30T17:22:38.048 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:18.0.0-6865.gfee4af45.el9.x86_64 30/33 2023-10-30T17:22:38.048 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:38.048 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-30T17:22:38.049 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T17:22:38.049 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T17:22:38.049 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:38.233 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-30T17:22:38.233 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:38.234 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:22:38.234 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:38.234 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:38.234 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:38.234 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:38.235 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:38.235 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:38.235 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-30T17:22:38.235 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-30T17:22:38.235 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-30T17:22:38.236 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-30T17:22:38.236 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-30T17:22:38.236 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-30T17:22:38.236 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-30T17:22:38.236 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-30T17:22:38.237 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-30T17:22:38.237 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-30T17:22:38.237 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-30T17:22:38.237 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-30T17:22:38.237 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-30T17:22:38.238 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-30T17:22:38.238 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-30T17:22:38.238 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2023-10-30T17:22:38.238 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-30T17:22:38.238 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-30T17:22:38.239 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-30T17:22:38.239 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-30T17:22:38.239 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-30T17:22:38.239 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-30T17:22:38.239 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-30T17:22:38.239 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-30T17:22:38.240 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-30T17:22:38.240 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-30T17:22:38.240 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-30T17:22:38.240 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:38.240 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:22:38.505 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:18.0.0-6865.gfee4af45.el9.x86_64 31/33 2023-10-30T17:22:38.529 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:18.0.0-6865.gfee4af45.el9.x86_64 31/33 2023-10-30T17:22:38.530 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:38.530 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-30T17:22:38.530 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T17:22:38.530 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T17:22:38.531 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:38.585 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base 2023-10-30T17:22:38.730 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:18.0.0-6865.gfee4af45.el9.x86_64 32/33 2023-10-30T17:22:38.754 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:18.0.0-6865.gfee4af45.el9.x86_64 32/33 2023-10-30T17:22:38.754 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:38.754 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-30T17:22:38.755 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T17:22:38.755 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T17:22:38.755 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:38.812 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:18.0.0-6865.gfee4af45.el9.x86_64 33/33 2023-10-30T17:22:39.136 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:53 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:22:39.228 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-base-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:22:39.277 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:22:39.278 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-30T17:22:39.278 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:22:39.336 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install cephadm 2023-10-30T17:22:39.879 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:53 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:22:40.020 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:22:40.021 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:40.021 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-30T17:22:40.022 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:40.022 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:22:40.022 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 270 k 2023-10-30T17:22:40.022 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:40.023 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:22:40.023 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:40.023 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:22:40.023 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:40.023 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 270 k 2023-10-30T17:22:40.024 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 265 k 2023-10-30T17:22:40.024 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:22:40.272 INFO:teuthology.orchestra.run.smithi170.stdout:cephadm-18.0.0-6865.gfee4af45.el9.noarch.rpm 1.1 MB/s | 270 kB 00:00 2023-10-30T17:22:40.273 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:40.273 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.0 MB/s | 270 kB 00:00 2023-10-30T17:22:40.274 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:22:40.278 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:22:40.279 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:22:40.281 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:22:40.282 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:22:40.355 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:22:40.416 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-2:18.0.0-6865.gfee4af45.el9.x86_64 33/33 2023-10-30T17:22:40.417 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:18.0.0-6865.gfee4af45.el9.x86_64 1/33 2023-10-30T17:22:40.417 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:18.0.0-6865.gfee4af45.el9.x86_64 2/33 2023-10-30T17:22:40.417 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:18.0.0-6865.gfee4af45.el9.x86_64 3/33 2023-10-30T17:22:40.417 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:18.0.0-6865.gfee4af45.el9.x86_64 4/33 2023-10-30T17:22:40.418 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:18.0.0-6865.gfee4af45.el9.x86_64 5/33 2023-10-30T17:22:40.418 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6865.gfee4af45.el9. 6/33 2023-10-30T17:22:40.418 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-30T17:22:40.418 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-30T17:22:40.418 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-30T17:22:40.419 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-30T17:22:40.419 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T17:22:40.419 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T17:22:40.419 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-30T17:22:40.420 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-30T17:22:40.420 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-30T17:22:40.420 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-30T17:22:40.420 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-30T17:22:40.420 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-30T17:22:40.420 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-30T17:22:40.421 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-30T17:22:40.421 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-30T17:22:40.421 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-30T17:22:40.421 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-30T17:22:40.421 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-30T17:22:40.422 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-30T17:22:40.422 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-30T17:22:40.422 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T17:22:40.422 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-30T17:22:40.423 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-30T17:22:40.423 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-30T17:22:40.424 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-30T17:22:40.424 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-30T17:22:40.812 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:22:40.901 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:22:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-30T17:22:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:41.227 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:22:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-30T17:22:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-30T17:22:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-30T17:22:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-30T17:22:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-30T17:22:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-30T17:22:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-30T17:22:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-30T17:22:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-30T17:22:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-30T17:22:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-30T17:22:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-30T17:22:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-30T17:22:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-30T17:22:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-30T17:22:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2023-10-30T17:22:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-30T17:22:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-30T17:22:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-30T17:22:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-30T17:22:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-30T17:22:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-30T17:22:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-30T17:22:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-30T17:22:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-30T17:22:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-30T17:22:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-30T17:22:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:41.235 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:22:41.267 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:22:41.540 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base 2023-10-30T17:22:41.556 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:22:41.556 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:41.556 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:22:41.557 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:41.557 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:41.557 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:22:41.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-immutable-object-cache 2023-10-30T17:22:42.093 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:50 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:22:42.185 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:22:42.234 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:22:42.235 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-30T17:22:42.235 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:22:42.292 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install cephadm 2023-10-30T17:22:42.328 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:56 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:22:42.472 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:22:42.473 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:42.473 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:22:42.473 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:42.473 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:22:42.474 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 142 k 2023-10-30T17:22:42.474 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:42.474 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:22:42.474 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:42.474 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:22:42.475 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:42.475 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 142 k 2023-10-30T17:22:42.475 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 400 k 2023-10-30T17:22:42.475 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:22:42.712 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-immutable-object-cache-18.0.0-6865.gfee4af 594 kB/s | 142 kB 00:00 2023-10-30T17:22:42.713 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:42.713 INFO:teuthology.orchestra.run.smithi170.stdout:Total 590 kB/s | 142 kB 00:00 2023-10-30T17:22:42.714 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:22:42.721 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:22:42.721 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:22:42.773 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:22:42.774 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:22:42.839 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:50 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:22:42.921 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:22:42.979 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:22:42.980 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:42.980 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2023-10-30T17:22:42.980 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:42.981 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:22:42.981 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 270 k 2023-10-30T17:22:42.981 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:42.981 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:22:42.981 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:42.982 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:22:42.982 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:42.982 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 270 k 2023-10-30T17:22:42.983 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 265 k 2023-10-30T17:22:42.983 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:22:43.013 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6865.gfee4af45. 1/1 2023-10-30T17:22:43.038 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6865.gfee4af45. 1/1 2023-10-30T17:22:43.039 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:43.039 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-30T17:22:43.039 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:43.212 INFO:teuthology.orchestra.run.smithi177.stdout:cephadm-18.0.0-6865.gfee4af45.el9.noarch.rpm 1.1 MB/s | 270 kB 00:00 2023-10-30T17:22:43.217 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:43.218 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.1 MB/s | 270 kB 00:00 2023-10-30T17:22:43.218 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:22:43.218 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:22:43.219 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:22:43.221 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:22:43.222 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:22:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:22:43.742 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:22:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:22:43.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6865.gfee4af45. 1/1 2023-10-30T17:22:43.979 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:43.979 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:22:43.980 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:43.980 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:43.980 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:22:44.169 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:22:44.218 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr 2023-10-30T17:22:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:22:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:44.443 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:22:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:44.444 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:22:44.654 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache 2023-10-30T17:22:44.770 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:58 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:22:44.865 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-mgr-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:22:44.913 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:22:44.914 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-30T17:22:44.914 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:22:44.985 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-dashboard 2023-10-30T17:22:45.210 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:53 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:22:45.358 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:22:45.359 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:45.359 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:22:45.359 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:45.360 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:22:45.360 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 142 k 2023-10-30T17:22:45.360 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:45.360 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:22:45.361 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:45.361 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:22:45.361 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:45.361 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 142 k 2023-10-30T17:22:45.361 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 400 k 2023-10-30T17:22:45.362 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:22:45.527 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:59 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:22:45.601 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-18.0.0-6865.gfee4af 586 kB/s | 142 kB 00:00 2023-10-30T17:22:45.602 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:45.602 INFO:teuthology.orchestra.run.smithi177.stdout:Total 583 kB/s | 142 kB 00:00 2023-10-30T17:22:45.602 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:22:45.609 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:22:45.609 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:22:45.661 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:22:45.661 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:22:45.667 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:22:45.668 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:45.668 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-30T17:22:45.669 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:45.669 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:22:45.669 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 1.8 M 2023-10-30T17:22:45.669 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-30T17:22:45.670 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 23 k 2023-10-30T17:22:45.670 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 15 k 2023-10-30T17:22:45.670 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-30T17:22:45.670 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-30T17:22:45.670 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-30T17:22:45.671 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2023-10-30T17:22:45.671 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-30T17:22:45.671 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:45.671 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:22:45.671 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:22:45.671 INFO:teuthology.orchestra.run.smithi170.stdout:Install 7 Packages 2023-10-30T17:22:45.672 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:45.672 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 2.1 M 2023-10-30T17:22:45.673 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 8.6 M 2023-10-30T17:22:45.673 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:22:45.802 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:22:45.868 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6865.gfee4af45. 1/1 2023-10-30T17:22:45.892 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6865.gfee4af45. 1/1 2023-10-30T17:22:45.893 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:22:45.893 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-30T17:22:45.893 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:46.041 INFO:teuthology.orchestra.run.smithi170.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6865.gfee 122 kB/s | 23 kB 00:00 2023-10-30T17:22:46.092 INFO:teuthology.orchestra.run.smithi170.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6865.gfee4 63 kB/s | 15 kB 00:00 2023-10-30T17:22:46.150 INFO:teuthology.orchestra.run.smithi170.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 83 kB/s | 9.0 kB 00:00 2023-10-30T17:22:46.267 INFO:teuthology.orchestra.run.smithi170.stdout:(4/7): ceph-mgr-dashboard-18.0.0-6865.gfee4af45 4.2 MB/s | 1.8 MB 00:00 2023-10-30T17:22:46.293 INFO:teuthology.orchestra.run.smithi170.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 216 kB/s | 31 kB 00:00 2023-10-30T17:22:46.318 INFO:teuthology.orchestra.run.smithi170.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 181 kB/s | 41 kB 00:00 2023-10-30T17:22:46.426 INFO:teuthology.orchestra.run.smithi170.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2023-10-30T17:22:46.428 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:46.428 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.7 MB/s | 2.1 MB 00:00 2023-10-30T17:22:46.429 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:22:46.436 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:22:46.437 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:22:46.504 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:22:46.505 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:22:46.817 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:22:46.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6865.gfee4af45. 1/1 2023-10-30T17:22:46.825 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:46.825 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:22:46.826 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:22:46.826 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:46.826 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:22:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-30T17:22:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-30T17:22:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-30T17:22:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-30T17:22:47.027 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6865.gfee4af45.el9.n 5/7 2023-10-30T17:22:47.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr 2023-10-30T17:22:47.268 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6865.gfee4af45.el9. 6/7 2023-10-30T17:22:47.305 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6865.gfee4af45.el9.noarc 7/7 2023-10-30T17:22:47.615 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:55 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:22:47.708 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-mgr-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:22:47.757 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:22:47.758 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-30T17:22:47.758 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:22:47.816 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-dashboard 2023-10-30T17:22:47.866 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6865.gfee4af45.el9.noarc 7/7 2023-10-30T17:22:47.866 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6865.gfee4af45.el9. 1/7 2023-10-30T17:22:47.866 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6865.gfee4af45.el9.noarc 2/7 2023-10-30T17:22:47.866 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6865.gfee4af45.el9.n 3/7 2023-10-30T17:22:47.866 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-30T17:22:47.867 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-30T17:22:47.867 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-30T17:22:48.180 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-30T17:22:48.180 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:48.181 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:22:48.181 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:48.181 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:48.181 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:48.181 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-30T17:22:48.182 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-30T17:22:48.182 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-30T17:22:48.182 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-30T17:22:48.182 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:48.182 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:22:48.359 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:56 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:22:48.401 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-30T17:22:48.516 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:22:48.517 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2023-10-30T17:22:48.517 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:48.517 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:22:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 1.8 M 2023-10-30T17:22:48.518 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2023-10-30T17:22:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 23 k 2023-10-30T17:22:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 15 k 2023-10-30T17:22:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-30T17:22:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-30T17:22:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-30T17:22:48.519 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2023-10-30T17:22:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-30T17:22:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:48.519 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:22:48.520 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:22:48.520 INFO:teuthology.orchestra.run.smithi177.stdout:Install 7 Packages 2023-10-30T17:22:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:48.521 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.1 M 2023-10-30T17:22:48.521 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 8.6 M 2023-10-30T17:22:48.521 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:22:48.944 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:02 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:22:49.085 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:22:49.086 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-10-30T17:22:49.086 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-30T17:22:49.087 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-10-30T17:22:49.087 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:22:49.087 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 7.4 M 2023-10-30T17:22:49.087 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-30T17:22:49.087 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-30T17:22:49.088 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-30T17:22:49.088 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-30T17:22:49.088 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-30T17:22:49.088 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-30T17:22:49.088 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-30T17:22:49.089 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-30T17:22:49.089 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-30T17:22:49.089 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-30T17:22:49.089 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-30T17:22:49.089 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-30T17:22:49.090 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:49.090 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:22:49.090 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-10-30T17:22:49.090 INFO:teuthology.orchestra.run.smithi170.stdout:Install 12 Packages 2023-10-30T17:22:49.090 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:49.091 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 37 M 2023-10-30T17:22:49.091 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 209 M 2023-10-30T17:22:49.092 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:22:49.171 INFO:teuthology.orchestra.run.smithi177.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6865.gfee 125 kB/s | 23 kB 00:00 2023-10-30T17:22:49.208 INFO:teuthology.orchestra.run.smithi170.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 265 kB/s | 32 kB 00:00 2023-10-30T17:22:49.222 INFO:teuthology.orchestra.run.smithi177.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6865.gfee4 64 kB/s | 15 kB 00:00 2023-10-30T17:22:49.242 INFO:teuthology.orchestra.run.smithi170.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 526 kB/s | 17 kB 00:00 2023-10-30T17:22:49.275 INFO:teuthology.orchestra.run.smithi170.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-30T17:22:49.280 INFO:teuthology.orchestra.run.smithi177.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 84 kB/s | 9.0 kB 00:00 2023-10-30T17:22:49.343 INFO:teuthology.orchestra.run.smithi170.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-10-30T17:22:49.389 INFO:teuthology.orchestra.run.smithi177.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 284 kB/s | 31 kB 00:00 2023-10-30T17:22:49.423 INFO:teuthology.orchestra.run.smithi177.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 204 kB/s | 41 kB 00:00 2023-10-30T17:22:49.468 INFO:teuthology.orchestra.run.smithi170.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2023-10-30T17:22:49.489 INFO:teuthology.orchestra.run.smithi177.stdout:(6/7): ceph-mgr-dashboard-18.0.0-6865.gfee4af45 3.5 MB/s | 1.8 MB 00:00 2023-10-30T17:22:49.493 INFO:teuthology.orchestra.run.smithi170.stdout:(6/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 205 kB 00:00 2023-10-30T17:22:49.565 INFO:teuthology.orchestra.run.smithi177.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2023-10-30T17:22:49.566 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:49.567 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.0 MB/s | 2.1 MB 00:01 2023-10-30T17:22:49.567 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:22:49.568 INFO:teuthology.orchestra.run.smithi170.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2023-10-30T17:22:49.574 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:22:49.575 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:22:49.594 INFO:teuthology.orchestra.run.smithi170.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-10-30T17:22:49.641 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:22:49.642 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:22:49.662 INFO:teuthology.orchestra.run.smithi170.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 12 MB/s | 803 kB 00:00 2023-10-30T17:22:49.836 INFO:teuthology.orchestra.run.smithi170.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-10-30T17:22:49.861 INFO:teuthology.orchestra.run.smithi170.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 986 kB/s | 195 kB 00:00 2023-10-30T17:22:49.946 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:22:50.016 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-30T17:22:50.050 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-30T17:22:50.104 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-30T17:22:50.136 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-30T17:22:50.165 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6865.gfee4af45.el9.n 5/7 2023-10-30T17:22:50.405 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6865.gfee4af45.el9. 6/7 2023-10-30T17:22:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6865.gfee4af45.el9.noarc 7/7 2023-10-30T17:22:50.645 INFO:teuthology.orchestra.run.smithi170.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 4.8 MB/s | 7.4 MB 00:01 2023-10-30T17:22:50.647 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:50.648 INFO:teuthology.orchestra.run.smithi170.stdout:Total 24 MB/s | 37 MB 00:01 2023-10-30T17:22:50.648 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:22:50.669 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:22:50.670 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:22:50.929 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6865.gfee4af45.el9.noarc 7/7 2023-10-30T17:22:50.930 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6865.gfee4af45.el9. 1/7 2023-10-30T17:22:50.930 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6865.gfee4af45.el9.noarc 2/7 2023-10-30T17:22:50.930 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6865.gfee4af45.el9.n 3/7 2023-10-30T17:22:50.930 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-30T17:22:50.931 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-30T17:22:50.931 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-30T17:22:51.106 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:22:51.106 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:22:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-30T17:22:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:51.251 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:22:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-30T17:22:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-30T17:22:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-30T17:22:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-30T17:22:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:51.253 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:22:51.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-30T17:22:51.818 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:22:51.891 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-30T17:22:51.924 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-30T17:22:51.987 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:59 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:22:52.015 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-30T17:22:52.061 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-30T17:22:52.127 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:22:52.128 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2023-10-30T17:22:52.128 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2023-10-30T17:22:52.129 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2023-10-30T17:22:52.129 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:22:52.129 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 7.4 M 2023-10-30T17:22:52.129 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2023-10-30T17:22:52.129 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-30T17:22:52.130 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-30T17:22:52.130 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-30T17:22:52.130 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-30T17:22:52.130 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-30T17:22:52.130 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-30T17:22:52.131 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-30T17:22:52.131 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-30T17:22:52.131 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-30T17:22:52.131 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-30T17:22:52.132 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-30T17:22:52.132 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:52.132 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:22:52.132 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2023-10-30T17:22:52.132 INFO:teuthology.orchestra.run.smithi177.stdout:Install 12 Packages 2023-10-30T17:22:52.133 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:22:52.133 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 37 M 2023-10-30T17:22:52.134 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 209 M 2023-10-30T17:22:52.134 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:22:52.246 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T17:22:52.247 INFO:teuthology.orchestra.run.smithi177.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 275 kB/s | 32 kB 00:00 2023-10-30T17:22:52.276 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T17:22:52.279 INFO:teuthology.orchestra.run.smithi177.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2023-10-30T17:22:52.313 INFO:teuthology.orchestra.run.smithi177.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-30T17:22:52.366 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-30T17:22:52.389 INFO:teuthology.orchestra.run.smithi177.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-10-30T17:22:52.489 INFO:teuthology.orchestra.run.smithi177.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2023-10-30T17:22:52.514 INFO:teuthology.orchestra.run.smithi177.stdout:(6/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 205 kB 00:00 2023-10-30T17:22:52.589 INFO:teuthology.orchestra.run.smithi177.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2023-10-30T17:22:52.614 INFO:teuthology.orchestra.run.smithi177.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-10-30T17:22:52.690 INFO:teuthology.orchestra.run.smithi177.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 11 MB/s | 803 kB 00:00 2023-10-30T17:22:52.857 INFO:teuthology.orchestra.run.smithi177.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-10-30T17:22:52.882 INFO:teuthology.orchestra.run.smithi177.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2023-10-30T17:22:53.004 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-30T17:22:53.120 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T17:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 7.0 MB/s | 7.4 MB 00:01 2023-10-30T17:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout:Total 35 MB/s | 37 MB 00:01 2023-10-30T17:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:22:53.631 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:22:53.631 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:22:54.328 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:22:54.352 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-30T17:22:54.410 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-30T17:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-30T17:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-30T17:22:54.573 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-30T17:22:54.757 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T17:22:54.788 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T17:22:54.879 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-30T17:22:55.393 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-30T17:22:55.418 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6865.gfee4a 12/12 2023-10-30T17:22:55.565 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-30T17:22:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T17:22:56.923 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-30T17:22:57.958 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-30T17:22:57.988 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6865.gfee4a 12/12 2023-10-30T17:22:58.007 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6865.gfee4a 12/12 2023-10-30T17:22:58.007 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6865.gfee4a 1/12 2023-10-30T17:22:58.007 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-30T17:22:58.007 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-30T17:22:58.007 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-30T17:22:58.008 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T17:22:58.008 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T17:22:58.008 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-30T17:22:58.008 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-30T17:22:58.009 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T17:22:58.009 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-30T17:22:58.009 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-30T17:22:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-30T17:22:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:58.449 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:22:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:22:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-30T17:22:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-30T17:22:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-30T17:22:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-30T17:22:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-30T17:22:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-30T17:22:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-30T17:22:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-30T17:22:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-30T17:22:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-30T17:22:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-30T17:22:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:58.451 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:22:58.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-rook 2023-10-30T17:22:59.329 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:13 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:22:59.471 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:22:59.472 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-10-30T17:22:59.472 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-30T17:22:59.472 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-10-30T17:22:59.472 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:22:59.473 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 48 k 2023-10-30T17:22:59.473 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-30T17:22:59.473 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-30T17:22:59.473 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-30T17:22:59.473 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-30T17:22:59.474 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-30T17:22:59.474 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-30T17:22:59.474 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-30T17:22:59.474 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-30T17:22:59.474 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-30T17:22:59.475 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-30T17:22:59.475 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-30T17:22:59.475 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-30T17:22:59.475 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-30T17:22:59.475 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:59.476 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:22:59.476 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-10-30T17:22:59.476 INFO:teuthology.orchestra.run.smithi170.stdout:Install 13 Packages 2023-10-30T17:22:59.476 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:22:59.477 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 2.0 M 2023-10-30T17:22:59.477 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 26 M 2023-10-30T17:22:59.477 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:22:59.944 INFO:teuthology.orchestra.run.smithi170.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 213 kB/s | 24 kB 00:00 2023-10-30T17:22:59.960 INFO:teuthology.orchestra.run.smithi170.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 138 kB/s | 18 kB 00:00 2023-10-30T17:23:00.002 INFO:teuthology.orchestra.run.smithi170.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2023-10-30T17:23:00.028 INFO:teuthology.orchestra.run.smithi170.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-10-30T17:23:00.053 INFO:teuthology.orchestra.run.smithi170.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-30T17:23:00.078 INFO:teuthology.orchestra.run.smithi170.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2023-10-30T17:23:00.104 INFO:teuthology.orchestra.run.smithi170.stdout:(7/13): ceph-mgr-rook-18.0.0-6865.gfee4af45.el9 174 kB/s | 48 kB 00:00 2023-10-30T17:23:00.229 INFO:teuthology.orchestra.run.smithi170.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 183 kB/s | 32 kB 00:00 2023-10-30T17:23:00.262 INFO:teuthology.orchestra.run.smithi170.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 81 kB/s | 15 kB 00:00 2023-10-30T17:23:00.329 INFO:teuthology.orchestra.run.smithi170.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 887 kB/s | 59 kB 00:00 2023-10-30T17:23:00.355 INFO:teuthology.orchestra.run.smithi170.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 961 kB/s | 197 kB 00:00 2023-10-30T17:23:00.388 INFO:teuthology.orchestra.run.smithi170.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 6.5 MB/s | 1.0 MB 00:00 2023-10-30T17:23:00.413 INFO:teuthology.orchestra.run.smithi170.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2023-10-30T17:23:00.417 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:00.418 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2023-10-30T17:23:00.418 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:23:00.430 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:23:00.430 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:23:00.464 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6865.gfee4a 12/12 2023-10-30T17:23:00.465 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6865.gfee4a 1/12 2023-10-30T17:23:00.465 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-30T17:23:00.465 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-30T17:23:00.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-30T17:23:00.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T17:23:00.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T17:23:00.467 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-30T17:23:00.467 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-30T17:23:00.467 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T17:23:00.468 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-30T17:23:00.468 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-30T17:23:00.527 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:23:00.528 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:23:00.893 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-30T17:23:00.893 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:00.893 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:00.894 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:23:00.894 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-30T17:23:00.894 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-30T17:23:00.894 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-30T17:23:00.894 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-30T17:23:00.895 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-30T17:23:00.895 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-30T17:23:00.895 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-30T17:23:00.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-30T17:23:00.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-30T17:23:00.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-30T17:23:00.896 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-30T17:23:00.896 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:00.896 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:01.056 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:23:01.241 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-rook 2023-10-30T17:23:01.252 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-30T17:23:01.306 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-30T17:23:01.361 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-30T17:23:01.399 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-30T17:23:01.440 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-30T17:23:01.534 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-30T17:23:01.635 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-30T17:23:01.692 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-30T17:23:01.796 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:09 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:01.937 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:01.938 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2023-10-30T17:23:01.938 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2023-10-30T17:23:01.939 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2023-10-30T17:23:01.939 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:01.939 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 48 k 2023-10-30T17:23:01.939 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2023-10-30T17:23:01.939 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-30T17:23:01.940 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-30T17:23:01.940 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-30T17:23:01.940 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-30T17:23:01.940 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-30T17:23:01.940 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-30T17:23:01.941 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-30T17:23:01.941 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-30T17:23:01.941 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-30T17:23:01.941 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-30T17:23:01.941 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-30T17:23:01.941 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-30T17:23:01.942 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:01.942 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:01.942 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2023-10-30T17:23:01.942 INFO:teuthology.orchestra.run.smithi177.stdout:Install 13 Packages 2023-10-30T17:23:01.943 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:01.943 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.0 M 2023-10-30T17:23:01.944 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 26 M 2023-10-30T17:23:01.944 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:23:02.104 INFO:teuthology.orchestra.run.smithi177.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 159 kB/s | 18 kB 00:00 2023-10-30T17:23:02.121 INFO:teuthology.orchestra.run.smithi177.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 187 kB/s | 24 kB 00:00 2023-10-30T17:23:02.162 INFO:teuthology.orchestra.run.smithi177.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2023-10-30T17:23:02.188 INFO:teuthology.orchestra.run.smithi177.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-10-30T17:23:02.213 INFO:teuthology.orchestra.run.smithi177.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-30T17:23:02.238 INFO:teuthology.orchestra.run.smithi177.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 861 kB/s | 43 kB 00:00 2023-10-30T17:23:02.288 INFO:teuthology.orchestra.run.smithi177.stdout:(7/13): ceph-mgr-rook-18.0.0-6865.gfee4af45.el9 160 kB/s | 48 kB 00:00 2023-10-30T17:23:02.406 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-30T17:23:02.447 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-30T17:23:02.481 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-30T17:23:02.527 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-30T17:23:02.530 INFO:teuthology.orchestra.run.smithi177.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2023-10-30T17:23:02.551 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:18.0.0-6865.gfee4af45.el9.noarch 13/13 2023-10-30T17:23:02.581 INFO:teuthology.orchestra.run.smithi177.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 44 kB/s | 15 kB 00:00 2023-10-30T17:23:02.714 INFO:teuthology.orchestra.run.smithi177.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 444 kB/s | 59 kB 00:00 2023-10-30T17:23:02.839 INFO:teuthology.orchestra.run.smithi177.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 390 kB/s | 197 kB 00:00 2023-10-30T17:23:02.856 INFO:teuthology.orchestra.run.smithi177.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 632 kB/s | 90 kB 00:00 2023-10-30T17:23:03.140 INFO:teuthology.orchestra.run.smithi177.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2023-10-30T17:23:03.144 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:03.144 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2023-10-30T17:23:03.144 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:23:03.156 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:23:03.157 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:23:03.253 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:23:03.254 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:23:03.376 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6865.gfee4af45.el9.noarch 13/13 2023-10-30T17:23:03.377 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6865.gfee4af45.el9.noarch 1/13 2023-10-30T17:23:03.377 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-30T17:23:03.377 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-30T17:23:03.377 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-30T17:23:03.378 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-30T17:23:03.378 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-30T17:23:03.378 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-30T17:23:03.378 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-30T17:23:03.378 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-30T17:23:03.379 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-30T17:23:03.379 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-30T17:23:03.379 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-30T17:23:03.767 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:23:03.769 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-30T17:23:03.769 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:03.769 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:23:03.769 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:23:03.769 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-30T17:23:03.770 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-30T17:23:03.770 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-30T17:23:03.770 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-30T17:23:03.770 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-30T17:23:03.770 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-30T17:23:03.771 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-30T17:23:03.771 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-30T17:23:03.771 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-30T17:23:03.771 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-30T17:23:03.771 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-30T17:23:03.772 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-30T17:23:03.772 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:03.772 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:03.946 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-30T17:23:04.007 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-30T17:23:04.011 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-cephadm 2023-10-30T17:23:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-30T17:23:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-30T17:23:04.141 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-30T17:23:04.234 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-30T17:23:04.335 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-30T17:23:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-30T17:23:04.563 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:18 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:04.705 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:04.706 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-10-30T17:23:04.706 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-30T17:23:04.706 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-10-30T17:23:04.706 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:23:04.707 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 132 k 2023-10-30T17:23:04.707 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-30T17:23:04.707 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-30T17:23:04.707 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-30T17:23:04.707 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-30T17:23:04.708 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-30T17:23:04.708 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-30T17:23:04.708 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:04.708 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:23:04.708 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-10-30T17:23:04.709 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-10-30T17:23:04.709 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:04.709 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 6.8 M 2023-10-30T17:23:04.710 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 33 M 2023-10-30T17:23:04.710 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:23:05.112 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-30T17:23:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-30T17:23:05.190 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-30T17:23:05.200 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-10-30T17:23:05.236 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-30T17:23:05.260 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:18.0.0-6865.gfee4af45.el9.noarch 13/13 2023-10-30T17:23:05.325 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2023-10-30T17:23:05.375 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6865.gfee4af45.e 409 kB/s | 132 kB 00:00 2023-10-30T17:23:05.417 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 629 kB/s | 58 kB 00:00 2023-10-30T17:23:05.451 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2023-10-30T17:23:05.476 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 507 kB/s | 51 kB 00:00 2023-10-30T17:23:05.478 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:05.478 INFO:teuthology.orchestra.run.smithi170.stdout:Total 8.8 MB/s | 6.8 MB 00:00 2023-10-30T17:23:05.479 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:23:05.487 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:23:05.488 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:23:05.576 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:23:05.577 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:23:05.879 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:23:06.042 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-30T17:23:06.056 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6865.gfee4af45.el9.noarch 13/13 2023-10-30T17:23:06.057 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6865.gfee4af45.el9.noarch 1/13 2023-10-30T17:23:06.057 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-30T17:23:06.057 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-30T17:23:06.057 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-30T17:23:06.058 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-30T17:23:06.058 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-30T17:23:06.058 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-30T17:23:06.058 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-30T17:23:06.058 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-30T17:23:06.059 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-30T17:23:06.059 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-30T17:23:06.059 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-30T17:23:06.094 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-30T17:23:06.432 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-30T17:23:06.432 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:06.432 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:06.433 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:23:06.433 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-30T17:23:06.433 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-30T17:23:06.433 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-30T17:23:06.433 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-30T17:23:06.434 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-30T17:23:06.434 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-30T17:23:06.434 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-30T17:23:06.434 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-30T17:23:06.434 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-30T17:23:06.435 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-30T17:23:06.435 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-30T17:23:06.435 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-30T17:23:06.435 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:06.435 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:06.605 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-30T17:23:06.661 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-cephadm 2023-10-30T17:23:06.679 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-30T17:23:06.740 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-30T17:23:06.765 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 6/6 2023-10-30T17:23:07.208 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:15 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:07.358 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:07.359 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2023-10-30T17:23:07.359 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2023-10-30T17:23:07.359 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2023-10-30T17:23:07.360 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:07.360 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 132 k 2023-10-30T17:23:07.360 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2023-10-30T17:23:07.360 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-30T17:23:07.360 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-30T17:23:07.361 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-30T17:23:07.361 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-30T17:23:07.361 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-30T17:23:07.361 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:07.362 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:07.362 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2023-10-30T17:23:07.362 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2023-10-30T17:23:07.363 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:07.364 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 6.8 M 2023-10-30T17:23:07.364 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 33 M 2023-10-30T17:23:07.365 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:23:07.530 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 6/6 2023-10-30T17:23:07.530 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 1/6 2023-10-30T17:23:07.530 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-30T17:23:07.530 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-30T17:23:07.531 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-30T17:23:07.531 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-30T17:23:07.836 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-30T17:23:07.836 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:07.837 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:23:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:23:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-30T17:23:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-30T17:23:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-30T17:23:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-30T17:23:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-30T17:23:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:07.838 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:07.866 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-10-30T17:23:07.983 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2023-10-30T17:23:08.008 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6865.gfee4af45.e 456 kB/s | 132 kB 00:00 2023-10-30T17:23:08.084 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-fuse 2023-10-30T17:23:08.325 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2023-10-30T17:23:08.342 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 153 kB/s | 51 kB 00:00 2023-10-30T17:23:08.484 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 888 kB/s | 548 kB 00:00 2023-10-30T17:23:08.485 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:08.486 INFO:teuthology.orchestra.run.smithi177.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2023-10-30T17:23:08.486 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:23:08.495 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:23:08.496 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:23:08.584 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:23:08.584 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:23:08.629 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:22 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:08.771 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:08.772 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:08.772 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-30T17:23:08.773 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:08.773 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:23:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 856 k 2023-10-30T17:23:08.773 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-30T17:23:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-30T17:23:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:08.774 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:23:08.774 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:08.774 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2023-10-30T17:23:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:08.775 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 934 k 2023-10-30T17:23:08.775 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.7 M 2023-10-30T17:23:08.775 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:23:08.883 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:23:08.886 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 698 kB/s | 79 kB 00:00 2023-10-30T17:23:09.046 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-30T17:23:09.101 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-30T17:23:09.103 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-fuse-18.0.0-6865.gfee4af45.el9.x86_ 2.5 MB/s | 856 kB 00:00 2023-10-30T17:23:09.104 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:09.105 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.8 MB/s | 934 kB 00:00 2023-10-30T17:23:09.105 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:23:09.124 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:23:09.125 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:23:09.199 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:23:09.199 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:23:09.376 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:23:09.520 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-30T17:23:09.586 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 2/2 2023-10-30T17:23:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-30T17:23:09.695 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-30T17:23:09.756 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-30T17:23:09.781 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 6/6 2023-10-30T17:23:10.263 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 2/2 2023-10-30T17:23:10.263 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 1/2 2023-10-30T17:23:10.602 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-30T17:23:10.602 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:10.603 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:23:10.603 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-30T17:23:10.603 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:10.603 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:10.632 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 6/6 2023-10-30T17:23:10.632 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 1/6 2023-10-30T17:23:10.632 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-30T17:23:10.632 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-30T17:23:10.633 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-30T17:23:10.633 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-30T17:23:10.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-volume 2023-10-30T17:23:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-30T17:23:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:10.939 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:23:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-30T17:23:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-30T17:23:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-30T17:23:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-30T17:23:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-30T17:23:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:10.941 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:11.188 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse 2023-10-30T17:23:11.436 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:25 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:11.582 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:11.584 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:11.584 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-30T17:23:11.584 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:11.584 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:23:11.584 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 262 k 2023-10-30T17:23:11.585 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:11.585 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:23:11.585 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:11.585 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:23:11.585 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:11.586 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 262 k 2023-10-30T17:23:11.586 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 1.2 M 2023-10-30T17:23:11.586 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:23:11.729 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:19 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:11.810 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-volume-18.0.0-6865.gfee4af45.el9.noarch.rp 1.1 MB/s | 262 kB 00:00 2023-10-30T17:23:11.811 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:11.811 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-10-30T17:23:11.811 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:23:11.818 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:23:11.819 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:23:11.853 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:23:11.853 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:23:11.870 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:11.871 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:11.871 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2023-10-30T17:23:11.871 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:11.872 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:11.872 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 856 k 2023-10-30T17:23:11.872 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2023-10-30T17:23:11.872 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-30T17:23:11.872 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:11.873 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:11.873 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:11.873 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2023-10-30T17:23:11.873 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:11.874 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 934 k 2023-10-30T17:23:11.874 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.7 M 2023-10-30T17:23:11.874 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:23:11.977 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 758 kB/s | 79 kB 00:00 2023-10-30T17:23:12.088 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:23:12.145 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:23:12.160 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:23:12.161 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:23:12.161 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-30T17:23:12.161 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:12.193 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-fuse-18.0.0-6865.gfee4af45.el9.x86_ 2.6 MB/s | 856 kB 00:00 2023-10-30T17:23:12.195 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:12.195 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.8 MB/s | 934 kB 00:00 2023-10-30T17:23:12.196 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:23:12.217 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:23:12.218 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:23:12.290 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:23:12.291 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:23:12.473 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:23:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-30T17:23:12.676 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 2/2 2023-10-30T17:23:13.219 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:23:13.220 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:13.220 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:23:13.220 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:23:13.221 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:13.221 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:13.376 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 2/2 2023-10-30T17:23:13.376 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 1/2 2023-10-30T17:23:13.548 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados-devel 2023-10-30T17:23:13.679 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-30T17:23:13.680 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:13.680 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:13.681 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-30T17:23:13.681 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:13.681 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:13.946 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-volume 2023-10-30T17:23:14.121 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:28 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:14.263 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:14.264 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:14.264 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:23:14.264 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:14.264 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:23:14.264 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 126 k 2023-10-30T17:23:14.265 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:14.265 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:23:14.265 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:14.265 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:23:14.265 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:14.266 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 126 k 2023-10-30T17:23:14.266 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 456 k 2023-10-30T17:23:14.266 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:23:14.499 INFO:teuthology.orchestra.run.smithi170.stdout:librados-devel-18.0.0-6865.gfee4af45.el9.x86_64 542 kB/s | 126 kB 00:00 2023-10-30T17:23:14.500 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:14.500 INFO:teuthology.orchestra.run.smithi170.stdout:Total 537 kB/s | 126 kB 00:00 2023-10-30T17:23:14.500 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:23:14.505 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:23:14.505 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:23:14.517 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:22 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:14.560 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:23:14.561 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:23:14.660 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:14.660 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:14.661 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2023-10-30T17:23:14.661 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:14.661 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:14.661 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:18.0.0-6865.gfee4af45.el9 ceph-noarch 262 k 2023-10-30T17:23:14.661 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:14.662 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:14.662 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:14.662 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:23:14.662 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:14.662 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 262 k 2023-10-30T17:23:14.663 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.2 M 2023-10-30T17:23:14.663 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:23:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:23:14.814 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:14.906 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-volume-18.0.0-6865.gfee4af45.el9.noarch.rp 1.0 MB/s | 262 kB 00:00 2023-10-30T17:23:14.907 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:14.907 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.0 MB/s | 262 kB 00:00 2023-10-30T17:23:14.907 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:23:14.914 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:23:14.915 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:23:14.948 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:23:14.948 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:23:15.166 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:23:15.224 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:23:15.240 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:23:15.241 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:23:15.241 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-30T17:23:15.241 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:15.260 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:15.570 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:15.570 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:15.570 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:23:15.571 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:15.571 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:15.571 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:15.829 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs2 2023-10-30T17:23:16.270 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:18.0.0-6865.gfee4af45.el9.noarch 1/1 2023-10-30T17:23:16.270 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:16.270 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:16.271 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:18.0.0-6865.gfee4af45.el9.noarch 2023-10-30T17:23:16.271 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:16.271 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:16.395 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:30 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:16.489 INFO:teuthology.orchestra.run.smithi170.stdout:Package libcephfs2-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:16.540 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:16.541 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-30T17:23:16.541 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:16.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel 2023-10-30T17:23:16.614 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs-devel 2023-10-30T17:23:17.141 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:25 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:17.158 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:31 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:17.282 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:23:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:17.283 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 126 k 2023-10-30T17:23:17.283 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:23:17.284 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 126 k 2023-10-30T17:23:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 456 k 2023-10-30T17:23:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:23:17.299 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:17.299 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:17.299 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:23:17.299 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:17.300 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:23:17.300 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 31 k 2023-10-30T17:23:17.300 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:17.300 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:23:17.300 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:17.301 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:23:17.301 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:17.301 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 31 k 2023-10-30T17:23:17.302 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 139 k 2023-10-30T17:23:17.302 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:23:17.477 INFO:teuthology.orchestra.run.smithi170.stdout:libcephfs-devel-18.0.0-6865.gfee4af45.el9.x86_6 176 kB/s | 31 kB 00:00 2023-10-30T17:23:17.478 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:17.479 INFO:teuthology.orchestra.run.smithi170.stdout:Total 174 kB/s | 31 kB 00:00 2023-10-30T17:23:17.479 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:23:17.482 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:23:17.482 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:23:17.509 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:23:17.510 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:23:17.519 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-18.0.0-6865.gfee4af45.el9.x86_64 536 kB/s | 126 kB 00:00 2023-10-30T17:23:17.520 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:17.520 INFO:teuthology.orchestra.run.smithi177.stdout:Total 532 kB/s | 126 kB 00:00 2023-10-30T17:23:17.521 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:23:17.526 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:23:17.526 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:23:17.582 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:23:17.582 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:23:17.590 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:23:17.709 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:17.738 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:23:17.844 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:18.093 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:18.287 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:18.376 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:18.376 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:18.377 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:23:18.377 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:18.377 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:18.377 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:18.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados2 2023-10-30T17:23:18.577 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:18.578 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:18.578 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:18.578 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:18.578 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:18.579 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:18.837 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2 2023-10-30T17:23:19.102 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:33 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:19.194 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:19.243 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:19.244 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-30T17:23:19.244 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:19.309 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librbd1 2023-10-30T17:23:19.404 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:27 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:19.500 INFO:teuthology.orchestra.run.smithi177.stdout:Package libcephfs2-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:19.551 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:19.552 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-30T17:23:19.553 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:19.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs-devel 2023-10-30T17:23:19.855 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:33 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:19.946 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:19.996 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:19.996 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-30T17:23:19.997 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:20.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rados 2023-10-30T17:23:20.154 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:28 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:20.294 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:20.295 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:20.296 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:23:20.296 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:20.296 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:20.296 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 31 k 2023-10-30T17:23:20.296 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:20.297 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:20.297 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:20.297 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:23:20.297 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:20.297 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 31 k 2023-10-30T17:23:20.298 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 139 k 2023-10-30T17:23:20.298 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:23:20.472 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs-devel-18.0.0-6865.gfee4af45.el9.x86_6 177 kB/s | 31 kB 00:00 2023-10-30T17:23:20.473 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:20.473 INFO:teuthology.orchestra.run.smithi177.stdout:Total 175 kB/s | 31 kB 00:00 2023-10-30T17:23:20.474 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:23:20.477 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:23:20.477 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:23:20.514 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:23:20.611 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:34 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:20.702 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rados-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:20.752 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:20.753 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-30T17:23:20.753 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:20.813 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rgw 2023-10-30T17:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:21.367 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:35 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:21.371 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:21.372 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:21.372 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:21.372 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:21.372 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:21.372 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:21.459 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rgw-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:21.509 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:21.510 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-30T17:23:21.510 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:21.556 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2 2023-10-30T17:23:21.575 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-cephfs 2023-10-30T17:23:22.100 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:30 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:22.116 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:36 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:22.190 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:22.209 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-cephfs-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:22.239 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:22.240 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-30T17:23:22.241 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:22.259 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:22.259 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-30T17:23:22.260 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:22.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1 2023-10-30T17:23:22.321 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rbd 2023-10-30T17:23:22.843 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:30 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:22.873 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:36 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:22.935 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:22.965 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rbd-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:22.984 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:22.985 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-30T17:23:22.986 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:23.015 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:23.016 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-30T17:23:23.016 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:23.044 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados 2023-10-30T17:23:23.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-fuse 2023-10-30T17:23:23.594 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:31 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:23.630 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:37 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:23.686 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rados-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:23.736 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:23.737 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-30T17:23:23.737 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:23.772 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:23.773 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:23.773 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-10-30T17:23:23.773 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:23.774 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:23:23.774 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 85 k 2023-10-30T17:23:23.774 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:23.774 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:23:23.774 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:23.775 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:23:23.775 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:23.775 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 85 k 2023-10-30T17:23:23.775 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 227 k 2023-10-30T17:23:23.775 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:23:23.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw 2023-10-30T17:23:23.981 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-fuse-18.0.0-6865.gfee4af45.el9.x86_64.rpm 411 kB/s | 85 kB 00:00 2023-10-30T17:23:23.981 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:23.982 INFO:teuthology.orchestra.run.smithi170.stdout:Total 408 kB/s | 85 kB 00:00 2023-10-30T17:23:23.982 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:23:23.988 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:23:23.989 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:23:24.043 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:23:24.043 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:23:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:23:24.278 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:24.345 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:32 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:24.437 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rgw-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:24.486 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:24.487 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-30T17:23:24.488 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:24.547 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs 2023-10-30T17:23:24.647 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:24.934 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:24.934 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:24.934 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:23:24.934 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:24.935 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:24.935 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:25.090 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:33 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:25.181 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-cephfs-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:25.194 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2023-10-30T17:23:25.230 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:25.231 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-30T17:23:25.231 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:25.283 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd 2023-10-30T17:23:25.757 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:39 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:25.835 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:33 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:25.905 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:25.905 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:25.905 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:23:25.906 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:25.906 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:23:25.906 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 3.0 M 2023-10-30T17:23:25.906 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:25.906 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:23:25.907 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:25.907 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:23:25.907 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:25.908 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.0 M 2023-10-30T17:23:25.908 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 12 M 2023-10-30T17:23:25.908 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:23:25.927 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rbd-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:23:25.977 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:25.978 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-30T17:23:25.978 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:26.031 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse 2023-10-30T17:23:26.409 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-mirror-18.0.0-6865.gfee4af45.el9.x86_64.rpm 6.0 MB/s | 3.0 MB 00:00 2023-10-30T17:23:26.410 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:26.410 INFO:teuthology.orchestra.run.smithi170.stdout:Total 6.0 MB/s | 3.0 MB 00:00 2023-10-30T17:23:26.410 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:23:26.419 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:23:26.419 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:23:26.491 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:23:26.492 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:23:26.578 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:34 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:26.720 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:26.721 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:26.721 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2023-10-30T17:23:26.721 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:26.722 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:26.722 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 85 k 2023-10-30T17:23:26.722 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:26.722 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:26.723 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:26.723 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:23:26.723 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:26.723 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 85 k 2023-10-30T17:23:26.723 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 227 k 2023-10-30T17:23:26.724 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:23:26.900 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:23:26.935 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-18.0.0-6865.gfee4af45.el9.x86_64.rpm 397 kB/s | 85 kB 00:00 2023-10-30T17:23:26.936 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:26.936 INFO:teuthology.orchestra.run.smithi177.stdout:Total 395 kB/s | 85 kB 00:00 2023-10-30T17:23:26.937 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:23:26.943 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:23:26.943 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:23:26.968 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:26.991 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:26.991 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:23:26.992 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-30T17:23:26.992 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T17:23:26.992 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T17:23:26.992 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:26.997 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:23:26.997 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:23:27.138 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:23:27.242 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:27.635 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:27.946 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:27.946 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:27.947 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:27.947 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:27.947 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:27.947 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:28.124 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:28.125 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:28.125 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:23:28.125 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:28.125 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:28.126 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:28.174 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror 2023-10-30T17:23:28.348 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-nbd 2023-10-30T17:23:28.736 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:36 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:28.880 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:28.881 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:23:28.882 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:28.882 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:28.882 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 3.0 M 2023-10-30T17:23:28.882 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:28.882 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:28.883 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:28.883 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:23:28.883 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:28.883 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 3.0 M 2023-10-30T17:23:28.883 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 12 M 2023-10-30T17:23:28.884 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:23:28.917 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:42 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:29.059 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:29.060 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-10-30T17:23:29.060 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:29.060 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:23:29.061 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 170 k 2023-10-30T17:23:29.061 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:29.061 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:23:29.061 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:29.061 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:23:29.062 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:29.062 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 170 k 2023-10-30T17:23:29.062 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 490 k 2023-10-30T17:23:29.062 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:23:29.296 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-nbd-18.0.0-6865.gfee4af45.el9.x86_64.rpm 722 kB/s | 170 kB 00:00 2023-10-30T17:23:29.297 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:29.297 INFO:teuthology.orchestra.run.smithi170.stdout:Total 717 kB/s | 170 kB 00:00 2023-10-30T17:23:29.297 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:23:29.305 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:23:29.305 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:23:29.360 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:23:29.361 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:23:29.396 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-18.0.0-6865.gfee4af45.el9.x86_64.rpm 5.9 MB/s | 3.0 MB 00:00 2023-10-30T17:23:29.397 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:29.397 INFO:teuthology.orchestra.run.smithi177.stdout:Total 5.9 MB/s | 3.0 MB 00:00 2023-10-30T17:23:29.398 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:23:29.406 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:23:29.406 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:23:29.479 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:23:29.479 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:23:29.508 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:23:29.627 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:29.896 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:23:29.947 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:29.970 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:29.971 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T17:23:29.971 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-30T17:23:29.971 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T17:23:29.971 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T17:23:29.972 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:30.130 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:30.408 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:30.409 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:30.409 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:23:30.409 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:30.410 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:30.410 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:30.639 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base-debuginfo 2023-10-30T17:23:31.110 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:31.111 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:31.111 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:31.111 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:31.111 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:31.112 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:31.192 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:45 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:31.338 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:31.339 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:31.339 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:23:31.340 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:31.340 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:23:31.340 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 74 M 2023-10-30T17:23:31.340 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-30T17:23:31.340 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 20 M 2023-10-30T17:23:31.341 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:31.341 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:23:31.341 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:31.341 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2023-10-30T17:23:31.341 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:31.341 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 94 M 2023-10-30T17:23:31.342 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 511 M 2023-10-30T17:23:31.342 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:23:31.343 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd 2023-10-30T17:23:31.901 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:39 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:32.044 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:32.045 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:32.045 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2023-10-30T17:23:32.045 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:32.046 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:32.046 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 170 k 2023-10-30T17:23:32.046 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:32.046 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:32.046 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:32.046 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:23:32.047 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:32.047 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 170 k 2023-10-30T17:23:32.047 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 490 k 2023-10-30T17:23:32.047 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:23:32.441 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-18.0.0-6865.gfee4af45.el9.x86_64.rpm 429 kB/s | 170 kB 00:00 2023-10-30T17:23:32.442 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:32.443 INFO:teuthology.orchestra.run.smithi177.stdout:Total 428 kB/s | 170 kB 00:00 2023-10-30T17:23:32.443 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:23:32.450 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:23:32.450 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:23:32.506 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:23:32.509 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:23:32.662 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:23:32.764 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:33.226 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:33.544 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:23:33.544 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:33.545 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:33.545 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:33.545 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:33.545 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:33.717 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): ceph-debuginfo-18.0.0-6865.gfee4af45.el9 8.5 MB/s | 20 MB 00:02 2023-10-30T17:23:33.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base-debuginfo 2023-10-30T17:23:34.335 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:42 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:34.479 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:34.480 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:34.480 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:23:34.481 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:34.481 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:34.481 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 74 M 2023-10-30T17:23:34.481 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2023-10-30T17:23:34.481 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 20 M 2023-10-30T17:23:34.482 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:34.482 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:34.482 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:34.482 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2023-10-30T17:23:34.482 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:34.483 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 94 M 2023-10-30T17:23:34.483 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 511 M 2023-10-30T17:23:34.483 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:23:35.758 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-base-debuginfo-18.0.0-6865.gfee4af4 17 MB/s | 74 MB 00:04 2023-10-30T17:23:35.761 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:35.761 INFO:teuthology.orchestra.run.smithi170.stdout:Total 21 MB/s | 94 MB 00:04 2023-10-30T17:23:35.761 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:23:35.766 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:23:35.766 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:23:36.260 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): ceph-debuginfo-18.0.0-6865.gfee4af45.el9 11 MB/s | 20 MB 00:01 2023-10-30T17:23:36.371 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:23:36.372 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:23:39.281 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:23:39.285 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-base-debuginfo-18.0.0-6865.gfee4af4 15 MB/s | 74 MB 00:04 2023-10-30T17:23:39.287 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:23:39.288 INFO:teuthology.orchestra.run.smithi177.stdout:Total 20 MB/s | 94 MB 00:04 2023-10-30T17:23:39.288 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:23:39.292 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:23:39.292 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:23:39.901 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:23:39.901 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:23:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:23:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/2 2023-10-30T17:23:47.041 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 2/2 2023-10-30T17:23:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/2 2023-10-30T17:23:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 2/2 2023-10-30T17:23:50.512 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 2/2 2023-10-30T17:23:50.513 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/2 2023-10-30T17:23:50.796 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2/2 2023-10-30T17:23:50.797 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:50.797 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:23:50.797 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:50.797 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:50.798 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:50.798 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:23:51.001 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-common-debuginfo 2023-10-30T17:23:51.563 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:05 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:23:51.715 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:23:51.716 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:51.716 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:23:51.716 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:51.717 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:23:51.717 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 411 M 2023-10-30T17:23:51.717 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:51.717 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:23:51.717 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:23:51.718 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:23:51.718 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:23:51.718 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 411 M 2023-10-30T17:23:51.718 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 1.7 G 2023-10-30T17:23:51.718 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:23:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 2/2 2023-10-30T17:23:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/2 2023-10-30T17:23:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2/2 2023-10-30T17:23:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:54.121 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:23:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:23:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:54.121 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:23:54.367 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-common-debuginfo 2023-10-30T17:23:54.913 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:02 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:23:55.054 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:23:55.055 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:23:55.056 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:55.056 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:23:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 411 M 2023-10-30T17:23:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:55.056 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:23:55.057 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:23:55.057 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:23:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:23:55.057 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 411 M 2023-10-30T17:23:55.057 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.7 G 2023-10-30T17:23:55.058 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:24:18.444 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-common-debuginfo-18.0.0-6865.gfee4af45.el9 15 MB/s | 411 MB 00:26 2023-10-30T17:24:18.444 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:24:18.445 INFO:teuthology.orchestra.run.smithi170.stdout:Total 15 MB/s | 411 MB 00:26 2023-10-30T17:24:18.445 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:24:18.449 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:24:18.450 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:24:20.915 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:24:20.915 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:24:23.836 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-common-debuginfo-18.0.0-6865.gfee4af45.el9 14 MB/s | 411 MB 00:28 2023-10-30T17:24:23.837 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:24:23.837 INFO:teuthology.orchestra.run.smithi177.stdout:Total 14 MB/s | 411 MB 00:28 2023-10-30T17:24:23.838 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:24:23.842 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:24:23.842 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:24:26.306 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:24:26.307 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:25:02.587 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:25:02.681 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:25:07.775 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:25:07.930 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:25:10.575 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:25:10.840 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:25:10.841 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:10.841 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:25:10.841 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:25:10.841 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:10.842 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:25:11.085 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-10-30T17:25:11.634 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:25 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:25:11.774 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:25:11.775 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-10-30T17:25:11.776 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:25:11.776 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-10-30T17:25:11.776 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:25:11.776 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 1.4 M 2023-10-30T17:25:11.777 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:11.777 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:25:11.777 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2023-10-30T17:25:11.777 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:25:11.777 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:11.778 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 1.4 M 2023-10-30T17:25:11.778 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 6.3 M 2023-10-30T17:25:11.778 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:25:12.842 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-68 1.3 MB/s | 1.4 MB 00:01 2023-10-30T17:25:12.842 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:25:12.843 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.3 MB/s | 1.4 MB 00:01 2023-10-30T17:25:12.843 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:25:12.847 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:25:12.847 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:25:12.892 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:25:12.893 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:25:13.196 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:25:13.298 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-6865. 1/1 2023-10-30T17:25:13.734 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-6865. 1/1 2023-10-30T17:25:13.989 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-6865. 1/1 2023-10-30T17:25:13.990 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:13.990 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:25:13.990 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:25:13.990 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:13.990 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:25:14.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw-debuginfo 2023-10-30T17:25:14.687 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:28 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:25:14.829 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:25:14.830 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:25:14.831 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:25:14.831 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:25:14.831 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:25:14.831 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 175 M 2023-10-30T17:25:14.832 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:14.832 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:25:14.832 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:25:14.832 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:25:14.832 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:14.833 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 175 M 2023-10-30T17:25:14.833 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 734 M 2023-10-30T17:25:14.833 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:25:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:25:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:25:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:15.858 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:25:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:25:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:15.859 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:25:16.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-10-30T17:25:16.646 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:03:24 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:25:16.798 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:25:16.799 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2023-10-30T17:25:16.799 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:25:16.799 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2023-10-30T17:25:16.799 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:25:16.800 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 1.4 M 2023-10-30T17:25:16.800 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:16.800 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:25:16.800 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2023-10-30T17:25:16.800 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:25:16.801 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:16.801 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 1.4 M 2023-10-30T17:25:16.801 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 6.3 M 2023-10-30T17:25:16.801 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:25:18.399 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-68 921 kB/s | 1.4 MB 00:01 2023-10-30T17:25:18.400 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:25:18.401 INFO:teuthology.orchestra.run.smithi177.stdout:Total 919 kB/s | 1.4 MB 00:01 2023-10-30T17:25:18.401 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:25:18.404 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:25:18.404 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:25:18.447 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:25:18.447 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:25:18.703 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:25:18.797 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-6865. 1/1 2023-10-30T17:25:19.193 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-6865. 1/1 2023-10-30T17:25:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-6865. 1/1 2023-10-30T17:25:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:19.457 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:25:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:25:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:19.457 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:25:19.654 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw-debuginfo 2023-10-30T17:25:20.201 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:03:28 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:25:20.345 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:25:20.346 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:25:20.346 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:25:20.347 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:25:20.347 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:25:20.347 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 175 M 2023-10-30T17:25:20.347 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:20.347 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:25:20.348 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:25:20.348 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:25:20.348 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:20.348 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 175 M 2023-10-30T17:25:20.348 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 734 M 2023-10-30T17:25:20.348 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:25:23.619 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-radosgw-debuginfo-18.0.0-6865.gfee4af45.el 20 MB/s | 175 MB 00:08 2023-10-30T17:25:23.620 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:25:23.621 INFO:teuthology.orchestra.run.smithi170.stdout:Total 20 MB/s | 175 MB 00:08 2023-10-30T17:25:23.621 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:25:23.625 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:25:23.625 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:25:24.688 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:25:24.689 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:25:29.710 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-radosgw-debuginfo-18.0.0-6865.gfee4af45.el 19 MB/s | 175 MB 00:09 2023-10-30T17:25:29.710 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:25:29.711 INFO:teuthology.orchestra.run.smithi177.stdout:Total 19 MB/s | 175 MB 00:09 2023-10-30T17:25:29.711 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:25:29.715 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:25:29.715 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:25:30.775 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:25:30.775 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:25:42.777 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:25:42.965 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x 1/1 2023-10-30T17:25:47.610 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x 1/1 2023-10-30T17:25:47.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x 1/1 2023-10-30T17:25:47.886 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:47.886 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:25:47.886 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:25:47.887 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:47.887 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:25:48.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-test-debuginfo 2023-10-30T17:25:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:25:48.619 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:04:02 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:25:48.646 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x 1/1 2023-10-30T17:25:48.761 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:25:48.762 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:25:48.763 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:25:48.763 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:25:48.763 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:25:48.763 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 627 M 2023-10-30T17:25:48.764 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:48.764 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:25:48.764 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:25:48.764 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:25:48.764 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:25:48.764 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 627 M 2023-10-30T17:25:48.765 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.8 G 2023-10-30T17:25:48.765 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:25:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x 1/1 2023-10-30T17:25:53.475 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x 1/1 2023-10-30T17:25:53.475 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:53.475 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:25:53.475 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:25:53.475 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:53.476 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:25:53.662 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test-debuginfo 2023-10-30T17:25:54.209 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:02 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:25:54.351 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:25:54.352 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:25:54.352 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:25:54.352 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:25:54.352 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:25:54.353 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 627 M 2023-10-30T17:25:54.353 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:54.353 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:25:54.353 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:25:54.353 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:25:54.354 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:25:54.354 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 627 M 2023-10-30T17:25:54.354 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.8 G 2023-10-30T17:25:54.354 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:26:27.649 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-test-debuginfo-18.0.0-6865.gfee4af45.el9.x 16 MB/s | 627 MB 00:38 2023-10-30T17:26:27.651 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:26:27.651 INFO:teuthology.orchestra.run.smithi170.stdout:Total 16 MB/s | 627 MB 00:38 2023-10-30T17:26:27.652 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:26:27.655 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:26:27.656 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:26:31.378 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:26:31.379 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:26:42.570 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-test-debuginfo-18.0.0-6865.gfee4af45.el9.x 13 MB/s | 627 MB 00:48 2023-10-30T17:26:42.570 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:26:42.571 INFO:teuthology.orchestra.run.smithi177.stdout:Total 13 MB/s | 627 MB 00:48 2023-10-30T17:26:42.571 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:26:42.576 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:26:42.576 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:26:46.288 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:26:46.289 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:27:37.046 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:27:37.155 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:27:46.385 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:27:46.655 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:27:46.655 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:27:46.656 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:27:46.656 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:27:46.656 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:27:46.656 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:27:46.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base-debuginfo 2023-10-30T17:27:47.451 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:06:01 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:27:47.544 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-base-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:27:47.595 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:27:47.596 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-10-30T17:27:47.596 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:27:47.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-debuginfo 2023-10-30T17:27:48.210 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:06:02 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:27:48.353 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:27:48.354 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:27:48.354 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:27:48.354 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:27:48.355 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:27:48.355 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 19 M 2023-10-30T17:27:48.355 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:27:48.355 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:27:48.355 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:27:48.356 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:27:48.356 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:27:48.356 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 19 M 2023-10-30T17:27:48.356 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 101 M 2023-10-30T17:27:48.356 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:27:50.692 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-mgr-debuginfo-18.0.0-6865.gfee4af45.el9.x8 8.2 MB/s | 19 MB 00:02 2023-10-30T17:27:50.693 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:27:50.693 INFO:teuthology.orchestra.run.smithi170.stdout:Total 8.2 MB/s | 19 MB 00:02 2023-10-30T17:27:50.693 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:27:50.697 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:27:50.697 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:27:50.844 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:27:50.844 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:27:51.665 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:27:52.226 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:27:53.026 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:27:53.140 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:27:54.114 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:27:54.379 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:27:54.380 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:27:54.380 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:27:54.380 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:27:54.380 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:27:54.381 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:27:54.551 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mds-debuginfo 2023-10-30T17:27:55.096 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:06:09 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:27:55.239 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:27:55.240 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:27:55.240 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:27:55.241 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:27:55.241 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:27:55.241 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 47 M 2023-10-30T17:27:55.241 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:27:55.241 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:27:55.242 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:27:55.242 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:27:55.242 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:27:55.242 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 47 M 2023-10-30T17:27:55.243 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 170 M 2023-10-30T17:27:55.243 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:27:58.720 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-mds-debuginfo-18.0.0-6865.gfee4af45.el9.x8 14 MB/s | 47 MB 00:03 2023-10-30T17:27:58.721 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:27:58.721 INFO:teuthology.orchestra.run.smithi170.stdout:Total 14 MB/s | 47 MB 00:03 2023-10-30T17:27:58.722 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:27:58.725 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:27:58.726 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:27:59.038 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:27:59.038 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:27:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:28:00.021 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:28:00.022 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:00.022 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:28:00.022 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:28:00.023 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:00.023 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:28:00.271 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base-debuginfo 2023-10-30T17:28:00.822 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:06:08 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:28:00.915 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 is already installed. 2023-10-30T17:28:00.965 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:28:00.966 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-30T17:28:00.967 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:28:01.035 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-debuginfo 2023-10-30T17:28:01.582 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:06:09 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:28:01.725 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:28:01.725 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:01.726 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:28:01.726 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:01.726 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:28:01.726 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 19 M 2023-10-30T17:28:01.727 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:01.727 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:28:01.727 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:01.727 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:28:01.727 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:01.728 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 19 M 2023-10-30T17:28:01.728 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 101 M 2023-10-30T17:28:01.728 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:28:03.714 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:28:03.804 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-mgr-debuginfo-18.0.0-6865.gfee4af45.el9.x8 9.3 MB/s | 19 MB 00:02 2023-10-30T17:28:03.804 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:28:03.805 INFO:teuthology.orchestra.run.smithi177.stdout:Total 9.3 MB/s | 19 MB 00:02 2023-10-30T17:28:03.805 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:28:03.808 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:28:03.809 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:28:03.901 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:03.955 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:28:03.955 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:28:05.314 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:05.578 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:05.579 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:28:05.579 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:28:05.579 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:28:05.580 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:28:05.580 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:28:05.749 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mon-debuginfo 2023-10-30T17:28:06.138 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:28:06.252 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:06.295 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:06:20 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:28:06.438 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:28:06.439 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:28:06.439 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:28:06.440 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:28:06.440 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:28:06.440 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 112 M 2023-10-30T17:28:06.440 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:28:06.440 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:28:06.441 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:28:06.441 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:28:06.441 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:28:06.441 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 112 M 2023-10-30T17:28:06.441 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 509 M 2023-10-30T17:28:06.442 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:28:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:07.452 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:07.453 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:07.453 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:28:07.453 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:28:07.453 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:07.454 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:28:07.620 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mds-debuginfo 2023-10-30T17:28:08.168 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:06:16 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:28:08.310 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:28:08.311 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:08.312 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:28:08.312 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:08.312 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:28:08.312 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 47 M 2023-10-30T17:28:08.312 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:08.313 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:28:08.313 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:08.313 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:28:08.313 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:08.313 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 47 M 2023-10-30T17:28:08.314 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 170 M 2023-10-30T17:28:08.314 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:28:11.922 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-mon-debuginfo-18.0.0-6865.gfee4af45.el9.x8 20 MB/s | 112 MB 00:05 2023-10-30T17:28:11.922 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:28:11.923 INFO:teuthology.orchestra.run.smithi170.stdout:Total 20 MB/s | 112 MB 00:05 2023-10-30T17:28:11.923 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:28:11.927 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:28:11.927 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:28:12.622 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:28:12.622 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:28:12.997 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-mds-debuginfo-18.0.0-6865.gfee4af45.el9.x8 10 MB/s | 47 MB 00:04 2023-10-30T17:28:12.998 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:28:12.999 INFO:teuthology.orchestra.run.smithi177.stdout:Total 10 MB/s | 47 MB 00:04 2023-10-30T17:28:12.999 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:28:13.003 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:28:13.003 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:28:13.314 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:28:13.314 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:28:17.967 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:28:18.129 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:19.522 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:19.782 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:19.783 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:28:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:28:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:19.783 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:28:19.959 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mon-debuginfo 2023-10-30T17:28:20.506 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:06:28 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:28:20.650 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:28:20.651 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:20.651 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:28:20.651 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:20.651 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:28:20.651 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 112 M 2023-10-30T17:28:20.651 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:20.652 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:28:20.652 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:20.652 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:28:20.652 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:20.652 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 112 M 2023-10-30T17:28:20.653 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 509 M 2023-10-30T17:28:20.653 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:28:24.361 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:28:24.479 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:25.740 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-mon-debuginfo-18.0.0-6865.gfee4af45.el9.x8 22 MB/s | 112 MB 00:05 2023-10-30T17:28:25.741 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:28:25.742 INFO:teuthology.orchestra.run.smithi177.stdout:Total 22 MB/s | 112 MB 00:05 2023-10-30T17:28:25.742 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:28:25.746 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:28:25.746 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:28:26.440 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:28:26.440 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:28:27.865 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:28.133 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:28.133 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:28:28.134 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:28:28.134 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:28:28.134 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:28:28.134 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:28:28.292 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-osd-debuginfo 2023-10-30T17:28:28.842 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:06:42 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:28:28.984 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:28:28.985 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:28:28.985 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:28:28.985 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:28:28.986 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:28:28.986 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 315 M 2023-10-30T17:28:28.986 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:28:28.986 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:28:28.986 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:28:28.987 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:28:28.987 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:28:28.987 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 315 M 2023-10-30T17:28:28.987 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 1.4 G 2023-10-30T17:28:28.987 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:28:38.130 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:28:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:41.508 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:41.786 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:28:41.786 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:41.786 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:28:41.787 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:28:41.787 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:41.787 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:28:41.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-osd-debuginfo 2023-10-30T17:28:42.504 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:06:50 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:28:42.646 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:28:42.647 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:42.647 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:28:42.647 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:42.647 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:28:42.648 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 315 M 2023-10-30T17:28:42.648 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:42.648 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:28:42.648 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:28:42.649 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:28:42.649 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:28:42.649 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 315 M 2023-10-30T17:28:42.649 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.4 G 2023-10-30T17:28:42.649 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:28:44.021 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-osd-debuginfo-18.0.0-6865.gfee4af45.el9.x8 21 MB/s | 315 MB 00:15 2023-10-30T17:28:44.022 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:28:44.023 INFO:teuthology.orchestra.run.smithi170.stdout:Total 21 MB/s | 315 MB 00:15 2023-10-30T17:28:44.023 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:28:44.027 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:28:44.027 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:28:45.909 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:28:45.909 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:28:56.556 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-osd-debuginfo-18.0.0-6865.gfee4af45.el9.x8 23 MB/s | 315 MB 00:13 2023-10-30T17:28:56.557 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:28:56.558 INFO:teuthology.orchestra.run.smithi177.stdout:Total 23 MB/s | 315 MB 00:13 2023-10-30T17:28:56.558 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:28:56.562 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:28:56.562 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:28:58.446 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:28:58.447 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:29:18.207 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:29:18.416 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:29:26.771 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:29:27.047 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:29:27.048 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:27.048 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:29:27.048 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:29:27.048 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:27.049 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:29:27.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-fuse-debuginfo 2023-10-30T17:29:27.775 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:07:41 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:29:27.915 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:29:27.916 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:27.917 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:29:27.917 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:27.917 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:29:27.917 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 5.8 M 2023-10-30T17:29:27.917 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:27.918 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:29:27.918 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:27.918 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:29:27.918 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:27.918 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 5.8 M 2023-10-30T17:29:27.919 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 28 M 2023-10-30T17:29:27.919 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:29:28.470 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-fuse-debuginfo-18.0.0-6865.gfee4af45.el9.x 10 MB/s | 5.8 MB 00:00 2023-10-30T17:29:28.471 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:29:28.471 INFO:teuthology.orchestra.run.smithi170.stdout:Total 10 MB/s | 5.8 MB 00:00 2023-10-30T17:29:28.471 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:29:28.474 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:29:28.474 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:29:28.539 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:29:28.540 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:29:29.255 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:29:29.359 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:29:29.900 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:29:30.172 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:29:30.173 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:30.173 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:29:30.173 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:29:30.173 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:30.173 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:29:30.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados-devel-debuginfo 2023-10-30T17:29:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:29:30.874 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:07:44 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:29:30.958 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:29:31.015 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:29:31.016 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:31.017 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:29:31.017 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:31.017 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:29:31.017 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 108 k 2023-10-30T17:29:31.017 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:31.018 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:29:31.018 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:31.018 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:29:31.018 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:31.018 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 108 k 2023-10-30T17:29:31.019 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 440 k 2023-10-30T17:29:31.019 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:29:31.522 INFO:teuthology.orchestra.run.smithi170.stdout:librados-devel-debuginfo-18.0.0-6865.gfee4af45. 215 kB/s | 108 kB 00:00 2023-10-30T17:29:31.523 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:29:31.523 INFO:teuthology.orchestra.run.smithi170.stdout:Total 214 kB/s | 108 kB 00:00 2023-10-30T17:29:31.524 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:29:31.526 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:29:31.527 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:29:31.560 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:29:31.560 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:29:31.678 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:29:31.779 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:29:32.134 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:29:32.397 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:29:32.397 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:32.397 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:29:32.397 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:29:32.398 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:32.398 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:29:32.552 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs2-debuginfo 2023-10-30T17:29:33.103 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:07:47 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:29:33.245 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:29:33.246 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:33.246 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:29:33.246 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:33.246 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:29:33.247 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 5.2 M 2023-10-30T17:29:33.247 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:33.247 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:29:33.247 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:33.247 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:29:33.247 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:33.248 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 5.2 M 2023-10-30T17:29:33.248 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 24 M 2023-10-30T17:29:33.248 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:29:34.459 INFO:teuthology.orchestra.run.smithi170.stdout:libcephfs2-debuginfo-18.0.0-6865.gfee4af45.el9. 4.2 MB/s | 5.2 MB 00:01 2023-10-30T17:29:34.460 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:29:34.460 INFO:teuthology.orchestra.run.smithi170.stdout:Total 4.2 MB/s | 5.2 MB 00:01 2023-10-30T17:29:34.461 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:29:34.465 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:29:34.465 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:29:34.532 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:29:34.533 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:29:35.181 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:29:35.300 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:29:35.811 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:29:36.082 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:29:36.082 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:36.082 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:29:36.083 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:29:36.083 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:36.083 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:29:36.259 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados2-debuginfo 2023-10-30T17:29:36.806 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:07:50 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:29:36.948 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:29:36.949 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:36.949 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:29:36.949 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:36.950 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:29:36.950 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 45 M 2023-10-30T17:29:36.950 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:36.950 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:29:36.950 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:36.951 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:29:36.951 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:36.951 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 45 M 2023-10-30T17:29:36.951 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 256 M 2023-10-30T17:29:36.951 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:29:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:29:39.251 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:29:39.252 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:39.252 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:29:39.252 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:29:39.252 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:39.253 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:29:39.378 INFO:teuthology.orchestra.run.smithi170.stdout:librados2-debuginfo-18.0.0-6865.gfee4af45.el9.x 19 MB/s | 45 MB 00:02 2023-10-30T17:29:39.379 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:29:39.380 INFO:teuthology.orchestra.run.smithi170.stdout:Total 19 MB/s | 45 MB 00:02 2023-10-30T17:29:39.380 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:29:39.384 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:29:39.384 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:29:39.456 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse-debuginfo 2023-10-30T17:29:39.698 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:29:39.698 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:29:40.002 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:07:48 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:29:40.143 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:29:40.144 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:40.144 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:29:40.145 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:40.145 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:29:40.145 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 5.8 M 2023-10-30T17:29:40.145 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:40.145 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:29:40.146 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:40.146 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:29:40.146 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:40.146 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 5.8 M 2023-10-30T17:29:40.146 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 28 M 2023-10-30T17:29:40.147 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:29:42.228 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-fuse-debuginfo-18.0.0-6865.gfee4af45.el9.x 2.8 MB/s | 5.8 MB 00:02 2023-10-30T17:29:42.229 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:29:42.229 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.8 MB/s | 5.8 MB 00:02 2023-10-30T17:29:42.230 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:29:42.233 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:29:42.234 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:29:42.304 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:29:42.305 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:29:43.014 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:29:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:29:43.694 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:29:43.975 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:29:43.976 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:43.976 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:29:43.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:29:43.976 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:43.977 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:29:44.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel-debuginfo 2023-10-30T17:29:44.676 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:07:52 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:29:44.817 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:29:44.818 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:44.818 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:29:44.818 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:44.819 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:29:44.819 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 108 k 2023-10-30T17:29:44.819 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:44.819 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:29:44.819 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:44.820 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:29:44.820 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:44.820 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 108 k 2023-10-30T17:29:44.820 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 440 k 2023-10-30T17:29:44.820 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:29:44.869 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:29:44.951 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:29:45.222 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-debuginfo-18.0.0-6865.gfee4af45. 269 kB/s | 108 kB 00:00 2023-10-30T17:29:45.223 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:29:45.223 INFO:teuthology.orchestra.run.smithi177.stdout:Total 268 kB/s | 108 kB 00:00 2023-10-30T17:29:45.224 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:29:45.226 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:29:45.227 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:29:45.260 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:29:45.260 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:29:45.373 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:29:45.471 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:29:45.833 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:29:46.100 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:29:46.100 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:46.100 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:29:46.100 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:29:46.100 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:46.101 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:29:46.271 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2-debuginfo 2023-10-30T17:29:46.816 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:07:54 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:29:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:29:46.958 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:29:46.959 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:29:46.959 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:46.959 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:29:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 5.2 M 2023-10-30T17:29:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:46.960 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:29:46.960 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:46.960 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:29:46.961 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:46.961 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 5.2 M 2023-10-30T17:29:46.961 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 24 M 2023-10-30T17:29:46.961 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:29:47.139 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:29:47.139 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:47.139 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:29:47.140 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:29:47.140 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:47.140 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:29:47.319 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librbd1-debuginfo 2023-10-30T17:29:47.859 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:08:01 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:29:48.000 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:29:48.001 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:48.001 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:29:48.001 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:48.001 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:29:48.002 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 51 M 2023-10-30T17:29:48.002 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:48.002 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:29:48.002 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:29:48.002 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:29:48.003 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:29:48.003 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 51 M 2023-10-30T17:29:48.003 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 298 M 2023-10-30T17:29:48.003 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:29:48.359 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs2-debuginfo-18.0.0-6865.gfee4af45.el9. 3.7 MB/s | 5.2 MB 00:01 2023-10-30T17:29:48.359 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:29:48.360 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.7 MB/s | 5.2 MB 00:01 2023-10-30T17:29:48.360 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:29:48.365 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:29:48.366 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:29:48.430 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:29:48.431 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:29:49.074 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:29:49.181 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:29:49.709 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:29:49.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:29:49.973 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:49.974 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:29:49.974 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:29:49.974 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:49.974 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:29:50.154 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2-debuginfo 2023-10-30T17:29:50.704 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:07:58 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:29:50.847 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:29:50.848 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:29:50.849 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:50.849 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:29:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 45 M 2023-10-30T17:29:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:50.849 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:29:50.850 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:29:50.850 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:29:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:29:50.850 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 45 M 2023-10-30T17:29:50.850 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 256 M 2023-10-30T17:29:50.851 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:29:53.280 INFO:teuthology.orchestra.run.smithi170.stdout:librbd1-debuginfo-18.0.0-6865.gfee4af45.el9.x86 9.6 MB/s | 51 MB 00:05 2023-10-30T17:29:53.281 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:29:53.282 INFO:teuthology.orchestra.run.smithi170.stdout:Total 9.6 MB/s | 51 MB 00:05 2023-10-30T17:29:53.282 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:29:53.286 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:29:53.286 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:29:53.630 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:29:53.631 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:29:54.902 INFO:teuthology.orchestra.run.smithi177.stdout:librados2-debuginfo-18.0.0-6865.gfee4af45.el9.x 11 MB/s | 45 MB 00:04 2023-10-30T17:29:54.903 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:29:54.904 INFO:teuthology.orchestra.run.smithi177.stdout:Total 11 MB/s | 45 MB 00:04 2023-10-30T17:29:54.904 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:29:54.908 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:29:54.908 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:29:55.218 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:29:55.218 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:29:59.496 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:29:59.762 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librbd1-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:30:00.449 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:30:01.915 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:02.194 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:02.195 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:02.195 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:30:02.195 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:02.195 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:02.196 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:30:02.301 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:30:02.370 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-cephfs-debuginfo 2023-10-30T17:30:02.562 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_ 1/1 2023-10-30T17:30:02.562 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:02.562 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:30:02.562 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:02.563 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:02.563 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:30:02.756 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1-debuginfo 2023-10-30T17:30:02.918 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:08:16 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:30:03.069 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:30:03.070 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:03.070 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:30:03.070 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:03.070 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:30:03.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 614 k 2023-10-30T17:30:03.071 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:03.071 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:30:03.071 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:03.071 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:30:03.072 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:03.072 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 614 k 2023-10-30T17:30:03.072 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.1 M 2023-10-30T17:30:03.072 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:30:03.304 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:11 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:30:03.333 INFO:teuthology.orchestra.run.smithi170.stdout:python3-cephfs-debuginfo-18.0.0-6865.gfee4af45. 2.3 MB/s | 614 kB 00:00 2023-10-30T17:30:03.333 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:03.334 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.3 MB/s | 614 kB 00:00 2023-10-30T17:30:03.334 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:30:03.337 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:30:03.337 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:30:03.374 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:30:03.375 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:30:03.446 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:30:03.447 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:03.447 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:30:03.447 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:03.447 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:30:03.447 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 51 M 2023-10-30T17:30:03.448 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:03.448 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:30:03.448 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:03.448 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:30:03.448 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:03.449 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 51 M 2023-10-30T17:30:03.449 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 298 M 2023-10-30T17:30:03.449 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:30:03.540 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:30:03.647 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:30:04.033 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:30:04.318 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:30:04.318 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:04.318 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:30:04.318 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:04.319 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:04.319 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:30:04.499 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rados-debuginfo 2023-10-30T17:30:05.043 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:08:19 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:30:05.186 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:30:05.187 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:05.187 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:30:05.188 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:05.188 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:30:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 770 k 2023-10-30T17:30:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:05.188 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:30:05.189 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:05.189 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:30:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:05.189 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 770 k 2023-10-30T17:30:05.189 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.7 M 2023-10-30T17:30:05.190 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:30:05.811 INFO:teuthology.orchestra.run.smithi170.stdout:python3-rados-debuginfo-18.0.0-6865.gfee4af45.e 1.2 MB/s | 770 kB 00:00 2023-10-30T17:30:05.812 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:05.812 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.2 MB/s | 770 kB 00:00 2023-10-30T17:30:05.812 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:30:05.815 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:30:05.815 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:30:05.846 INFO:teuthology.orchestra.run.smithi177.stdout:librbd1-debuginfo-18.0.0-6865.gfee4af45.el9.x86 21 MB/s | 51 MB 00:02 2023-10-30T17:30:05.847 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:05.848 INFO:teuthology.orchestra.run.smithi177.stdout:Total 21 MB/s | 51 MB 00:02 2023-10-30T17:30:05.848 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:30:05.852 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:30:05.852 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:30:05.853 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:30:05.853 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:30:06.028 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:30:06.126 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-debuginfo-2:18.0.0-6865.gfee4af45.el9. 1/1 2023-10-30T17:30:06.196 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:30:06.197 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:30:06.541 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-6865.gfee4af45.el9. 1/1 2023-10-30T17:30:06.799 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-6865.gfee4af45.el9. 1/1 2023-10-30T17:30:06.800 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:06.800 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:30:06.800 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:06.800 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:06.801 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:30:06.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rbd-debuginfo 2023-10-30T17:30:07.523 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:08:21 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:30:07.666 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:30:07.667 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:07.667 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:30:07.667 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:07.668 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:30:07.668 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 1.1 M 2023-10-30T17:30:07.668 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:07.668 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:30:07.668 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:07.669 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:30:07.669 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:07.669 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 1.1 M 2023-10-30T17:30:07.669 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 4.3 M 2023-10-30T17:30:07.669 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:30:07.964 INFO:teuthology.orchestra.run.smithi170.stdout:python3-rbd-debuginfo-18.0.0-6865.gfee4af45.el9 3.8 MB/s | 1.1 MB 00:00 2023-10-30T17:30:07.965 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:07.965 INFO:teuthology.orchestra.run.smithi170.stdout:Total 3.8 MB/s | 1.1 MB 00:00 2023-10-30T17:30:07.966 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:30:07.968 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:30:07.969 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:30:08.008 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:30:08.009 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:30:08.225 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:30:08.330 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:09.075 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:09.075 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:09.076 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:30:09.076 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:09.076 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:09.076 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:30:09.249 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rgw-debuginfo 2023-10-30T17:30:09.798 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:08:23 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:30:09.939 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:30:09.940 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:09.941 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:30:09.941 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:09.941 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:30:09.941 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 337 k 2023-10-30T17:30:09.941 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:09.941 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:30:09.942 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:09.942 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:30:09.942 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:09.942 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 337 k 2023-10-30T17:30:09.942 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 1.0 M 2023-10-30T17:30:09.943 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:30:10.342 INFO:teuthology.orchestra.run.smithi170.stdout:python3-rgw-debuginfo-18.0.0-6865.gfee4af45.el9 840 kB/s | 337 kB 00:00 2023-10-30T17:30:10.343 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:10.344 INFO:teuthology.orchestra.run.smithi170.stdout:Total 837 kB/s | 337 kB 00:00 2023-10-30T17:30:10.344 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:30:10.347 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:30:10.347 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:30:10.383 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:30:10.383 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:30:10.520 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:30:10.624 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:10.996 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:11.291 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:30:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:11.292 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:30:11.463 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-fuse-debuginfo 2023-10-30T17:30:12.014 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:08:26 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:30:12.077 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:30:12.157 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:30:12.158 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:12.158 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:30:12.159 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:12.159 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:30:12.159 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 506 k 2023-10-30T17:30:12.159 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:12.159 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:30:12.160 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:12.160 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:30:12.160 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:12.160 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 506 k 2023-10-30T17:30:12.160 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.5 M 2023-10-30T17:30:12.161 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:30:12.161 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librbd1-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:12.954 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-fuse-debuginfo-18.0.0-6865.gfee4af45.el9.x8 637 kB/s | 506 kB 00:00 2023-10-30T17:30:12.955 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:12.955 INFO:teuthology.orchestra.run.smithi170.stdout:Total 635 kB/s | 506 kB 00:00 2023-10-30T17:30:12.955 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:30:12.958 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:30:12.959 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:30:12.995 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:30:12.996 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:30:13.157 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:30:13.285 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:30:13.657 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:30:13.934 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:30:13.934 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:13.934 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:30:13.934 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:13.935 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:13.935 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:30:14.110 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror-debuginfo 2023-10-30T17:30:14.278 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:14.536 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:14.537 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:14.537 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:30:14.537 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:14.537 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:14.538 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:30:14.657 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:08:28 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:30:14.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs-debuginfo 2023-10-30T17:30:14.799 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:30:14.800 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:14.801 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:30:14.801 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:14.801 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:30:14.801 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 51 M 2023-10-30T17:30:14.801 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:14.802 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:30:14.802 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:14.802 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:30:14.802 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:14.803 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 51 M 2023-10-30T17:30:14.803 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 289 M 2023-10-30T17:30:14.803 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:30:15.270 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:23 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:30:15.411 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:30:15.411 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:15.412 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:30:15.412 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:15.412 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:30:15.412 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 614 k 2023-10-30T17:30:15.412 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:15.413 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:30:15.413 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:15.413 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:30:15.413 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:15.413 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 614 k 2023-10-30T17:30:15.414 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.1 M 2023-10-30T17:30:15.414 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:30:15.798 INFO:teuthology.orchestra.run.smithi177.stdout:python3-cephfs-debuginfo-18.0.0-6865.gfee4af45. 1.6 MB/s | 614 kB 00:00 2023-10-30T17:30:15.799 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:15.799 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.6 MB/s | 614 kB 00:00 2023-10-30T17:30:15.800 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:30:15.802 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:30:15.803 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:30:15.839 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:30:15.839 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:30:16.002 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:30:16.123 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:30:16.582 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:30:16.857 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-6865.gfee4af45.el9 1/1 2023-10-30T17:30:16.858 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:16.858 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:30:16.858 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:16.858 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:16.859 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:30:17.065 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados-debuginfo 2023-10-30T17:30:17.607 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:25 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:30:17.748 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:30:17.749 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:17.749 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:30:17.749 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:17.749 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:30:17.749 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 770 k 2023-10-30T17:30:17.750 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:17.750 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:30:17.750 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:17.750 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:30:17.750 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:17.751 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 770 k 2023-10-30T17:30:17.751 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.7 M 2023-10-30T17:30:17.751 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:30:18.085 INFO:teuthology.orchestra.run.smithi177.stdout:python3-rados-debuginfo-18.0.0-6865.gfee4af45.e 2.2 MB/s | 770 kB 00:00 2023-10-30T17:30:18.085 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:18.086 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.2 MB/s | 770 kB 00:00 2023-10-30T17:30:18.086 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:30:18.089 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:30:18.089 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:30:18.107 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-mirror-debuginfo-18.0.0-6865.gfee4af45.el9. 15 MB/s | 51 MB 00:03 2023-10-30T17:30:18.108 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:18.108 INFO:teuthology.orchestra.run.smithi170.stdout:Total 15 MB/s | 51 MB 00:03 2023-10-30T17:30:18.109 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:30:18.113 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:30:18.113 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:30:18.127 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:30:18.127 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:30:18.310 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:30:18.425 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-debuginfo-2:18.0.0-6865.gfee4af45.el9. 1/1 2023-10-30T17:30:18.446 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:30:18.447 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:30:18.788 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-6865.gfee4af45.el9. 1/1 2023-10-30T17:30:19.052 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-6865.gfee4af45.el9. 1/1 2023-10-30T17:30:19.053 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:19.053 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:30:19.053 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:19.053 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:19.054 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:30:19.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd-debuginfo 2023-10-30T17:30:19.794 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:27 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:30:19.937 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:30:19.938 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:19.938 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:30:19.938 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:19.939 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:30:19.939 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 1.1 M 2023-10-30T17:30:19.939 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:19.939 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:30:19.939 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:19.940 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:30:19.940 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:19.940 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 1.1 M 2023-10-30T17:30:19.940 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 4.3 M 2023-10-30T17:30:19.940 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:30:20.296 INFO:teuthology.orchestra.run.smithi177.stdout:python3-rbd-debuginfo-18.0.0-6865.gfee4af45.el9 3.2 MB/s | 1.1 MB 00:00 2023-10-30T17:30:20.297 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:20.297 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.1 MB/s | 1.1 MB 00:00 2023-10-30T17:30:20.297 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:30:20.300 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:30:20.300 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:30:20.339 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:30:20.339 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:30:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:30:20.668 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:21.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:21.352 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:21.352 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:30:21.352 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:21.353 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:30:21.593 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw-debuginfo 2023-10-30T17:30:22.141 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:30 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:30:22.283 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:30:22.284 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:30:22.285 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:22.285 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:30:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 337 k 2023-10-30T17:30:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:22.285 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:30:22.286 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:22.286 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:30:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:22.286 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 337 k 2023-10-30T17:30:22.286 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.0 M 2023-10-30T17:30:22.287 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:30:22.624 INFO:teuthology.orchestra.run.smithi177.stdout:python3-rgw-debuginfo-18.0.0-6865.gfee4af45.el9 994 kB/s | 337 kB 00:00 2023-10-30T17:30:22.625 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:22.626 INFO:teuthology.orchestra.run.smithi177.stdout:Total 990 kB/s | 337 kB 00:00 2023-10-30T17:30:22.626 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:30:22.628 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:30:22.628 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:30:22.662 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:30:22.663 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:30:22.798 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:30:22.906 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:23.276 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:23.533 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x8 1/1 2023-10-30T17:30:23.533 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:23.533 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:30:23.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:23.534 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:23.534 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:30:23.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse-debuginfo 2023-10-30T17:30:24.235 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:30:24.270 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:32 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:30:24.329 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:30:24.412 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:30:24.413 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:30:24.414 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:24.414 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:30:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 506 k 2023-10-30T17:30:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:24.414 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:30:24.415 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:24.415 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:30:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:24.415 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 506 k 2023-10-30T17:30:24.415 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.5 M 2023-10-30T17:30:24.416 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:30:25.260 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-debuginfo-18.0.0-6865.gfee4af45.el9.x8 598 kB/s | 506 kB 00:00 2023-10-30T17:30:25.261 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:25.261 INFO:teuthology.orchestra.run.smithi177.stdout:Total 597 kB/s | 506 kB 00:00 2023-10-30T17:30:25.262 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:30:25.265 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:30:25.265 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:30:25.302 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:30:25.303 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:30:25.462 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:30:25.583 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:30:25.971 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:30:26.249 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_6 1/1 2023-10-30T17:30:26.249 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:26.250 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:30:26.250 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:26.250 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:26.250 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:30:26.329 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:30:26.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror-debuginfo 2023-10-30T17:30:26.597 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:30:26.598 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:26.598 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:30:26.598 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:26.598 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:26.599 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:30:26.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-nbd-debuginfo 2023-10-30T17:30:26.988 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:34 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:30:27.128 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:30:27.129 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:27.129 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:30:27.130 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:27.130 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:30:27.130 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 51 M 2023-10-30T17:30:27.130 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:27.131 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:30:27.131 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:27.131 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:30:27.131 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:27.131 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 51 M 2023-10-30T17:30:27.132 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 289 M 2023-10-30T17:30:27.132 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:30:27.320 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:08:41 ago on Mon 30 Oct 2023 05:21:46 PM UTC. 2023-10-30T17:30:27.462 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-30T17:30:27.462 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:27.463 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-10-30T17:30:27.463 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:27.463 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-30T17:30:27.464 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 1.2 M 2023-10-30T17:30:27.464 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:27.464 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-30T17:30:27.465 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-30T17:30:27.465 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-10-30T17:30:27.466 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:27.466 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 1.2 M 2023-10-30T17:30:27.466 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 5.4 M 2023-10-30T17:30:27.467 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-30T17:30:28.034 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-nbd-debuginfo-18.0.0-6865.gfee4af45.el9.x86 2.2 MB/s | 1.2 MB 00:00 2023-10-30T17:30:28.035 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:28.035 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.2 MB/s | 1.2 MB 00:00 2023-10-30T17:30:28.036 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-30T17:30:28.038 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-30T17:30:28.038 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-30T17:30:28.079 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-30T17:30:28.080 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-30T17:30:28.315 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-30T17:30:28.432 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:28.853 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:29.114 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:29.114 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:29.115 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-30T17:30:29.115 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:29.115 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:29.115 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-30T17:30:29.289 DEBUG:teuthology.parallel:result is None 2023-10-30T17:30:30.063 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-debuginfo-18.0.0-6865.gfee4af45.el9. 17 MB/s | 51 MB 00:02 2023-10-30T17:30:30.063 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:30.064 INFO:teuthology.orchestra.run.smithi177.stdout:Total 17 MB/s | 51 MB 00:02 2023-10-30T17:30:30.064 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:30:30.068 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:30:30.069 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:30:30.404 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:30:30.405 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:30:36.219 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:30:36.302 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:30:38.274 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:30:38.548 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86 1/1 2023-10-30T17:30:38.549 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:38.549 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:30:38.549 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:38.550 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:38.550 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:30:38.771 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd-debuginfo 2023-10-30T17:30:39.318 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:47 ago on Mon 30 Oct 2023 05:21:52 PM UTC. 2023-10-30T17:30:39.459 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-30T17:30:39.460 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:39.460 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2023-10-30T17:30:39.460 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:39.460 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2023-10-30T17:30:39.460 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-6865.gfee4af45.el9 ceph 1.2 M 2023-10-30T17:30:39.461 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:39.461 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2023-10-30T17:30:39.461 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2023-10-30T17:30:39.461 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2023-10-30T17:30:39.461 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:39.462 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 1.2 M 2023-10-30T17:30:39.462 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 5.4 M 2023-10-30T17:30:39.462 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2023-10-30T17:30:40.523 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-debuginfo-18.0.0-6865.gfee4af45.el9.x86 1.2 MB/s | 1.2 MB 00:01 2023-10-30T17:30:40.523 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2023-10-30T17:30:40.524 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.2 MB/s | 1.2 MB 00:01 2023-10-30T17:30:40.524 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2023-10-30T17:30:40.527 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2023-10-30T17:30:40.527 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2023-10-30T17:30:40.569 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2023-10-30T17:30:40.570 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2023-10-30T17:30:40.802 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2023-10-30T17:30:40.913 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:41.320 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:41.598 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 1/1 2023-10-30T17:30:41.598 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:41.598 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2023-10-30T17:30:41.599 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-debuginfo-2:18.0.0-6865.gfee4af45.el9.x86_64 2023-10-30T17:30:41.599 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:41.599 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-30T17:30:41.814 DEBUG:teuthology.parallel:result is None 2023-10-30T17:30:41.815 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fee4af45ddecf08e014b4503e218408d9e5c5be2 2023-10-30T17:30:41.967 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-30T17:30:41.991 INFO:teuthology.orchestra.run.smithi170.stdout:18.0.0-6865.gfee4af45.el9 2023-10-30T17:30:41.992 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6865.gfee4af45.el9 2023-10-30T17:30:41.992 INFO:teuthology.task.install:The correct ceph version 18.0.0-6865.gfee4af45 is installed. 2023-10-30T17:30:41.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fee4af45ddecf08e014b4503e218408d9e5c5be2 2023-10-30T17:30:42.143 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-30T17:30:42.166 INFO:teuthology.orchestra.run.smithi177.stdout:18.0.0-6865.gfee4af45.el9 2023-10-30T17:30:42.167 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6865.gfee4af45.el9 2023-10-30T17:30:42.167 INFO:teuthology.task.install:The correct ceph version 18.0.0-6865.gfee4af45 is installed. 2023-10-30T17:30:42.168 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T17:30:42.168 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:30:42.169 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T17:30:42.201 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:30:42.201 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T17:30:42.234 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T17:30:42.234 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:30:42.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T17:30:42.272 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T17:30:42.342 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:30:42.342 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T17:30:42.369 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T17:30:42.435 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T17:30:42.436 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:30:42.436 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T17:30:42.467 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T17:30:42.538 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:30:42.539 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T17:30:42.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T17:30:42.631 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T17:30:42.632 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:30:42.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T17:30:42.661 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T17:30:42.730 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:30:42.731 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T17:30:42.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T17:30:42.824 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T17:30:42.892 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T17:30:42.893 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /var/log/ceph 2023-10-30T17:30:42.895 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2023-10-30T17:30:42.922 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T17:30:42.923 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T17:30:42.965 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T17:30:42.991 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T17:30:42.992 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T17:30:43.034 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T17:30:43.071 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T17:30:43.072 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T17:30:43.072 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201', 'branch': 'wip-yuri10-testing-2023-10-27-0804', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7441201', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri10-testing-2023-10-27-0804'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'fee4af45ddecf08e014b4503e218408d9e5c5be2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'fee4af45ddecf08e014b4503e218408d9e5c5be2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri10-testing-2023-10-27-0804', 'sha1': 'fee4af45ddecf08e014b4503e218408d9e5c5be2'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 5679, 'sha1': 'fee4af45ddecf08e014b4503e218408d9e5c5be2', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri10-testing-2023-10-27-0804', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'fee4af45ddecf08e014b4503e218408d9e5c5be2', 'targets': {'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEsA2T3Z7BhEScoWd2R+zeQ3a4mkh6Qpyg5hRdYQ+wKe/dOBtKMtrJRAdobB9gMKKCdoByietgpKaqqfGE15ufg=', 'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKldg/iMffn8uQIfGj3LsSXEf/wPuI0S90sdsB603K8/X9pi5GwOwl1hilfS5DV06t4Vp5SxbwhF17bGKJ+zSg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-30_15:34:36', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T17:30:43.073 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T17:30:43.103 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T17:30:43.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T17:30:43.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T17:30:43.185 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:30:43.185 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T17:30:43.239 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T17:30:43.240 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2023-10-30T17:30:43.294 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T17:30:43.295 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T17:30:43.295 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-10-30T17:30:43.295 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T17:30:43.295 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T17:30:43.296 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-30 17:23:27.233422234 +0000 2023-10-30T17:30:43.296 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-30 17:21:00.603224813 +0000 2023-10-30T17:30:43.296 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-30 17:21:00.603224813 +0000 2023-10-30T17:30:43.296 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-10-30 17:21:00.603224813 +0000 2023-10-30T17:30:43.297 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T17:30:43.361 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-30T17:30:43.361 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-30T17:30:43.361 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.3002e-05 s, 6.2 MB/s 2023-10-30T17:30:43.362 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T17:30:43.418 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2023-10-30T17:30:43.473 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T17:30:43.473 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T17:30:43.473 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-10-30T17:30:43.473 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T17:30:43.474 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T17:30:43.474 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-30 17:23:27.233422234 +0000 2023-10-30T17:30:43.474 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-30 17:21:01.075214081 +0000 2023-10-30T17:30:43.474 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-30 17:21:01.075214081 +0000 2023-10-30T17:30:43.475 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-10-30 17:21:01.075214081 +0000 2023-10-30T17:30:43.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T17:30:43.538 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-30T17:30:43.539 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-30T17:30:43.539 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.9701e-05 s, 5.1 MB/s 2023-10-30T17:30:43.540 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T17:30:43.595 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2023-10-30T17:30:43.649 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T17:30:43.649 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T17:30:43.650 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-10-30T17:30:43.650 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T17:30:43.650 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T17:30:43.650 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-30 17:23:27.234422211 +0000 2023-10-30T17:30:43.650 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-30 17:21:01.550203280 +0000 2023-10-30T17:30:43.651 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-30 17:21:01.550203280 +0000 2023-10-30T17:30:43.651 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-10-30 17:21:01.550203280 +0000 2023-10-30T17:30:43.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T17:30:43.714 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-30T17:30:43.714 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-30T17:30:43.714 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.7557e-05 s, 5.2 MB/s 2023-10-30T17:30:43.715 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T17:30:43.771 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2023-10-30T17:30:43.825 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T17:30:43.826 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T17:30:43.826 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-10-30T17:30:43.826 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T17:30:43.826 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T17:30:43.826 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-30 17:23:27.234422211 +0000 2023-10-30T17:30:43.827 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-30 17:21:02.031192342 +0000 2023-10-30T17:30:43.827 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-30 17:21:02.031192342 +0000 2023-10-30T17:30:43.827 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-10-30 17:21:02.031192342 +0000 2023-10-30T17:30:43.827 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T17:30:43.890 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-30T17:30:43.890 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-30T17:30:43.891 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 7.8339e-05 s, 6.5 MB/s 2023-10-30T17:30:43.892 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T17:30:43.947 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T17:30:43.947 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:30:43.948 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T17:30:43.964 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T17:30:43.964 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2023-10-30T17:30:44.020 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T17:30:44.020 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T17:30:44.020 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-10-30T17:30:44.021 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T17:30:44.021 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T17:30:44.021 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2023-10-30 17:23:30.194053217 +0000 2023-10-30T17:30:44.021 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2023-10-30 17:20:57.468581163 +0000 2023-10-30T17:30:44.021 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2023-10-30 17:20:57.468581163 +0000 2023-10-30T17:30:44.021 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2023-10-30 17:20:57.468581163 +0000 2023-10-30T17:30:44.022 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T17:30:44.086 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2023-10-30T17:30:44.087 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2023-10-30T17:30:44.087 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000210031 s, 2.4 MB/s 2023-10-30T17:30:44.088 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T17:30:44.144 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2023-10-30T17:30:44.199 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T17:30:44.199 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T17:30:44.199 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-10-30T17:30:44.200 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T17:30:44.200 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T17:30:44.200 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2023-10-30 17:23:30.195053190 +0000 2023-10-30T17:30:44.200 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2023-10-30 17:20:58.004566610 +0000 2023-10-30T17:30:44.200 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2023-10-30 17:20:58.004566610 +0000 2023-10-30T17:30:44.201 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2023-10-30 17:20:58.004566610 +0000 2023-10-30T17:30:44.201 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T17:30:44.265 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2023-10-30T17:30:44.265 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2023-10-30T17:30:44.266 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000222336 s, 2.3 MB/s 2023-10-30T17:30:44.267 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T17:30:44.323 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2023-10-30T17:30:44.378 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T17:30:44.378 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T17:30:44.378 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-10-30T17:30:44.378 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T17:30:44.379 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T17:30:44.379 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2023-10-30 17:23:30.195053190 +0000 2023-10-30T17:30:44.379 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2023-10-30 17:20:58.505553009 +0000 2023-10-30T17:30:44.379 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2023-10-30 17:20:58.505553009 +0000 2023-10-30T17:30:44.379 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2023-10-30 17:20:58.505553009 +0000 2023-10-30T17:30:44.380 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T17:30:44.443 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2023-10-30T17:30:44.444 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2023-10-30T17:30:44.444 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000226443 s, 2.3 MB/s 2023-10-30T17:30:44.445 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T17:30:44.500 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2023-10-30T17:30:44.555 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T17:30:44.555 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T17:30:44.555 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-10-30T17:30:44.555 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T17:30:44.556 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T17:30:44.556 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2023-10-30 17:23:30.195053190 +0000 2023-10-30T17:30:44.556 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2023-10-30 17:20:59.010539298 +0000 2023-10-30T17:30:44.556 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2023-10-30 17:20:59.010539298 +0000 2023-10-30T17:30:44.556 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2023-10-30 17:20:59.010539298 +0000 2023-10-30T17:30:44.557 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T17:30:44.621 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2023-10-30T17:30:44.621 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2023-10-30T17:30:44.621 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000188555 s, 2.7 MB/s 2023-10-30T17:30:44.622 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T17:30:44.678 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T17:30:44.678 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T17:30:44.679 INFO:tasks.ceph:Generating config... 2023-10-30T17:30:44.680 INFO:tasks.ceph:[client] debug monc = 20 2023-10-30T17:30:44.680 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T17:30:44.681 INFO:tasks.ceph:[client] debug objecter = 20 2023-10-30T17:30:44.681 INFO:tasks.ceph:[client] debug rados = 20 2023-10-30T17:30:44.681 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-30T17:30:44.682 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-10-30T17:30:44.682 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-30T17:30:44.682 INFO:tasks.ceph:[global] ms type = async 2023-10-30T17:30:44.682 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-10-30T17:30:44.683 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T17:30:44.683 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T17:30:44.683 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T17:30:44.683 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T17:30:44.684 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T17:30:44.684 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-10-30T17:30:44.684 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-10-30T17:30:44.685 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-10-30T17:30:44.685 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-10-30T17:30:44.685 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-10-30T17:30:44.685 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-10-30T17:30:44.686 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-10-30T17:30:44.686 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-10-30T17:30:44.686 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-30T17:30:44.687 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-30T17:30:44.687 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-30T17:30:44.687 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T17:30:44.687 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T17:30:44.688 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-10-30T17:30:44.688 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-10-30T17:30:44.688 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-10-30T17:30:44.689 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-30T17:30:44.689 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T17:30:44.689 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T17:30:44.689 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-30T17:30:44.690 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T17:30:44.690 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T17:30:44.690 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T17:30:44.691 INFO:tasks.ceph:[osd] osd class default list = * 2023-10-30T17:30:44.691 INFO:tasks.ceph:[osd] osd class load list = * 2023-10-30T17:30:44.691 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-10-30T17:30:44.691 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-10-30T17:30:44.692 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-10-30T17:30:44.692 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T17:30:44.692 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-10-30T17:30:44.692 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-10-30T17:30:44.693 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-10-30T17:30:44.693 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T17:30:44.693 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-10-30T17:30:44.693 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-10-30T17:30:44.694 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-30T17:30:44.694 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T17:30:44.694 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T17:30:44.754 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T17:30:44.757 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T17:30:44.801 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T17:30:44.868 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.170'), ('mon.c', '[v2:172.21.15.170:3301,v1:172.21.15.170:6790]'), ('mon.b', '172.21.15.177')] 2023-10-30T17:30:44.869 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.170,[v2:172.21.15.170:3301,v1:172.21.15.170:6790],172.21.15.177', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-10-30T17:30:44.870 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:30:44.870 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T17:30:44.923 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.170 --addv c '[v2:172.21.15.170:3301,v1:172.21.15.170:6790]' --add b 172.21.15.177 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:30:45.007 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:30:45.007 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: generated fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:30:45.008 INFO:teuthology.orchestra.run.smithi170.stdout:setting min_mon_release = pacific 2023-10-30T17:30:45.008 INFO:teuthology.orchestra.run.smithi170.stdout:epoch 0 2023-10-30T17:30:45.008 INFO:teuthology.orchestra.run.smithi170.stdout:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:30:45.008 INFO:teuthology.orchestra.run.smithi170.stdout:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:30:45.008 INFO:teuthology.orchestra.run.smithi170.stdout:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:30:45.009 INFO:teuthology.orchestra.run.smithi170.stdout:min_mon_release 16 (pacific) 2023-10-30T17:30:45.009 INFO:teuthology.orchestra.run.smithi170.stdout:election_strategy: 1 2023-10-30T17:30:45.009 INFO:teuthology.orchestra.run.smithi170.stdout:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:30:45.009 INFO:teuthology.orchestra.run.smithi170.stdout:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:30:45.009 INFO:teuthology.orchestra.run.smithi170.stdout:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:30:45.010 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T17:30:45.011 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T17:30:45.025 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID dbedaa2b-1947-40d2-95bd-9b4fc7298a39... 2023-10-30T17:30:45.026 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T17:30:45.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T17:30:45.115 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2023-10-30T17:30:45.115 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2023-10-30T17:30:45.115 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T17:30:45.116 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2023-10-30T17:30:45.116 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.116 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2023-10-30T17:30:45.116 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.116 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2023-10-30T17:30:45.117 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.117 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2023-10-30T17:30:45.117 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2023-10-30T17:30:45.117 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.117 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2023-10-30T17:30:45.118 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2023-10-30T17:30:45.118 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.118 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T17:30:45.118 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2023-10-30T17:30:45.118 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.119 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T17:30:45.119 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T17:30:45.119 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.119 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2023-10-30T17:30:45.119 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.120 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2023-10-30T17:30:45.120 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2023-10-30T17:30:45.120 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2023-10-30T17:30:45.120 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2023-10-30T17:30:45.120 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2023-10-30T17:30:45.120 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2023-10-30T17:30:45.121 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2023-10-30T17:30:45.121 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T17:30:45.121 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T17:30:45.121 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2023-10-30T17:30:45.121 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.122 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T17:30:45.122 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.122 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2023-10-30T17:30:45.122 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.122 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2023-10-30T17:30:45.123 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.123 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2023-10-30T17:30:45.123 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2023-10-30T17:30:45.123 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2023-10-30T17:30:45.123 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = "172.21.15.170,[v2:172.21.15.170:3301,v1:172.21.15.170:6790],172.21.15.177" 2023-10-30T17:30:45.124 INFO:teuthology.orchestra.run.smithi170.stdout: mon client directed command retry = 5 2023-10-30T17:30:45.124 INFO:teuthology.orchestra.run.smithi170.stdout: mon election default strategy = 1 2023-10-30T17:30:45.124 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject socket failures = 5000 2023-10-30T17:30:45.124 INFO:teuthology.orchestra.run.smithi170.stdout: ms type = async 2023-10-30T17:30:45.124 INFO:teuthology.orchestra.run.smithi170.stdout: osd heartbeat grace = 80 2023-10-30T17:30:45.124 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:30:45.125 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.125 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2023-10-30T17:30:45.125 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2023-10-30T17:30:45.125 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.125 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2023-10-30T17:30:45.126 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2023-10-30T17:30:45.126 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock profile = high_recovery_ops 2023-10-30T17:30:45.126 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.126 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2023-10-30T17:30:45.126 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2023-10-30T17:30:45.127 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.127 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2023-10-30T17:30:45.127 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2023-10-30T17:30:45.127 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2023-10-30T17:30:45.127 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.127 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2023-10-30T17:30:45.128 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2023-10-30T17:30:45.128 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.128 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2023-10-30T17:30:45.128 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.128 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2023-10-30T17:30:45.129 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.129 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2023-10-30T17:30:45.129 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.129 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2023-10-30T17:30:45.129 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2023-10-30T17:30:45.130 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.130 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2023-10-30T17:30:45.130 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2023-10-30T17:30:45.130 INFO:teuthology.orchestra.run.smithi170.stdout: bdev async discard = True 2023-10-30T17:30:45.130 INFO:teuthology.orchestra.run.smithi170.stdout: bdev enable discard = True 2023-10-30T17:30:45.131 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore allocator = bitmap 2023-10-30T17:30:45.131 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2023-10-30T17:30:45.131 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2023-10-30T17:30:45.131 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore zero block detection = True 2023-10-30T17:30:45.131 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 1 2023-10-30T17:30:45.131 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 1 2023-10-30T17:30:45.132 INFO:teuthology.orchestra.run.smithi170.stdout: debug monc = 20 2023-10-30T17:30:45.132 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-30T17:30:45.132 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2023-10-30T17:30:45.132 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 10 2023-10-30T17:30:45.132 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T17:30:45.133 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2023-10-30T17:30:45.133 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2023-10-30T17:30:45.133 INFO:teuthology.orchestra.run.smithi170.stdout: osd class default list = * 2023-10-30T17:30:45.133 INFO:teuthology.orchestra.run.smithi170.stdout: osd class load list = * 2023-10-30T17:30:45.133 INFO:teuthology.orchestra.run.smithi170.stdout: osd client watch timeout = 120 2023-10-30T17:30:45.134 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify cached snaps = True 2023-10-30T17:30:45.134 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify missing on start = True 2023-10-30T17:30:45.134 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2023-10-30T17:30:45.134 INFO:teuthology.orchestra.run.smithi170.stdout: osd fast shutdown = False 2023-10-30T17:30:45.134 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock override recovery settings = True 2023-10-30T17:30:45.135 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2023-10-30T17:30:45.135 INFO:teuthology.orchestra.run.smithi170.stdout: osd op queue = debug_random 2023-10-30T17:30:45.135 INFO:teuthology.orchestra.run.smithi170.stdout: osd op queue cut off = debug_random 2023-10-30T17:30:45.135 INFO:teuthology.orchestra.run.smithi170.stdout: osd shutdown pgref assert = True 2023-10-30T17:30:45.135 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.135 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2023-10-30T17:30:45.136 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-30T17:30:45.136 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2023-10-30T17:30:45.136 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-10-30T17:30:45.136 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-10-30T17:30:45.136 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-10-30T17:30:45.137 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-10-30T17:30:45.137 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2023-10-30T17:30:45.137 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.137 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2023-10-30T17:30:45.137 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-30T17:30:45.138 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-10-30T17:30:45.138 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2023-10-30T17:30:45.138 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-10-30T17:30:45.138 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2023-10-30T17:30:45.138 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2023-10-30T17:30:45.138 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-10-30T17:30:45.139 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2023-10-30T17:30:45.139 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2023-10-30T17:30:45.139 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-10-30T17:30:45.139 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.139 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T17:30:45.140 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T17:30:45.140 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T17:30:45.140 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.140 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T17:30:45.140 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T17:30:45.140 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T17:30:45.141 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.141 INFO:teuthology.orchestra.run.smithi170.stdout: # 1m isn't quite enough 2023-10-30T17:30:45.141 INFO:teuthology.orchestra.run.smithi170.stdout: mon_down_mkfs_grace = 2m 2023-10-30T17:30:45.141 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.141 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_filestore_osds = false 2023-10-30T17:30:45.142 INFO:teuthology.orchestra.run.smithi170.stdout: mon min osdmap epochs = 50 2023-10-30T17:30:45.142 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd crush smoke test = False 2023-10-30T17:30:45.142 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune interval = 2 2023-10-30T17:30:45.142 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune min = 15 2023-10-30T17:30:45.142 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune txsize = 2 2023-10-30T17:30:45.143 INFO:teuthology.orchestra.run.smithi170.stdout: mon scrub interval = 300 2023-10-30T17:30:45.143 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on pool no app = False 2023-10-30T17:30:45.143 INFO:teuthology.orchestra.run.smithi170.stdout: paxos service trim min = 10 2023-10-30T17:30:45.143 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:30:45.143 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2023-10-30T17:30:45.143 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2023-10-30T17:30:45.144 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2023-10-30T17:30:45.144 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2023-10-30T17:30:45.144 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T17:30:45.144 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T17:30:45.144 INFO:teuthology.orchestra.run.smithi170.stdout: debug monc = 20 2023-10-30T17:30:45.145 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-30T17:30:45.145 INFO:teuthology.orchestra.run.smithi170.stdout: debug objecter = 20 2023-10-30T17:30:45.145 INFO:teuthology.orchestra.run.smithi170.stdout: debug rados = 20 2023-10-30T17:30:45.145 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2023-10-30T17:30:45.145 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.c] 2023-10-30T17:30:45.146 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.b] 2023-10-30T17:30:45.147 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2023-10-30T17:30:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2023-10-30T17:30:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T17:30:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2023-10-30T17:30:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2023-10-30T17:30:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2023-10-30T17:30:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2023-10-30T17:30:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2023-10-30T17:30:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2023-10-30T17:30:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2023-10-30T17:30:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T17:30:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2023-10-30T17:30:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T17:30:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T17:30:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2023-10-30T17:30:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2023-10-30T17:30:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2023-10-30T17:30:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2023-10-30T17:30:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2023-10-30T17:30:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2023-10-30T17:30:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2023-10-30T17:30:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2023-10-30T17:30:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T17:30:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T17:30:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2023-10-30T17:30:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T17:30:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2023-10-30T17:30:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2023-10-30T17:30:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2023-10-30T17:30:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2023-10-30T17:30:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2023-10-30T17:30:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.170,[v2:172.21.15.170:3301,v1:172.21.15.170:6790],172.21.15.177" 2023-10-30T17:30:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: mon client directed command retry = 5 2023-10-30T17:30:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: mon election default strategy = 1 2023-10-30T17:30:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject socket failures = 5000 2023-10-30T17:30:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: ms type = async 2023-10-30T17:30:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: osd heartbeat grace = 80 2023-10-30T17:30:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:30:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.157 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2023-10-30T17:30:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2023-10-30T17:30:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2023-10-30T17:30:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2023-10-30T17:30:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2023-10-30T17:30:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2023-10-30T17:30:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2023-10-30T17:30:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2023-10-30T17:30:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2023-10-30T17:30:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2023-10-30T17:30:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2023-10-30T17:30:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2023-10-30T17:30:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2023-10-30T17:30:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2023-10-30T17:30:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2023-10-30T17:30:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2023-10-30T17:30:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2023-10-30T17:30:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2023-10-30T17:30:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2023-10-30T17:30:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2023-10-30T17:30:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2023-10-30T17:30:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore allocator = bitmap 2023-10-30T17:30:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2023-10-30T17:30:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2023-10-30T17:30:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore zero block detection = True 2023-10-30T17:30:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1 2023-10-30T17:30:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1 2023-10-30T17:30:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2023-10-30T17:30:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2023-10-30T17:30:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2023-10-30T17:30:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 10 2023-10-30T17:30:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T17:30:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2023-10-30T17:30:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2023-10-30T17:30:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: osd class default list = * 2023-10-30T17:30:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: osd class load list = * 2023-10-30T17:30:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: osd client watch timeout = 120 2023-10-30T17:30:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify cached snaps = True 2023-10-30T17:30:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify missing on start = True 2023-10-30T17:30:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2023-10-30T17:30:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: osd fast shutdown = False 2023-10-30T17:30:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock override recovery settings = True 2023-10-30T17:30:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2023-10-30T17:30:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue = debug_random 2023-10-30T17:30:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue cut off = debug_random 2023-10-30T17:30:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: osd shutdown pgref assert = True 2023-10-30T17:30:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.168 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2023-10-30T17:30:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2023-10-30T17:30:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2023-10-30T17:30:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2023-10-30T17:30:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2023-10-30T17:30:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2023-10-30T17:30:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2023-10-30T17:30:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2023-10-30T17:30:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.169 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2023-10-30T17:30:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2023-10-30T17:30:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2023-10-30T17:30:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2023-10-30T17:30:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2023-10-30T17:30:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2023-10-30T17:30:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2023-10-30T17:30:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2023-10-30T17:30:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2023-10-30T17:30:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2023-10-30T17:30:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2023-10-30T17:30:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T17:30:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T17:30:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T17:30:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T17:30:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T17:30:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T17:30:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2023-10-30T17:30:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2023-10-30T17:30:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2023-10-30T17:30:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: mon min osdmap epochs = 50 2023-10-30T17:30:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd crush smoke test = False 2023-10-30T17:30:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune interval = 2 2023-10-30T17:30:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune min = 15 2023-10-30T17:30:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune txsize = 2 2023-10-30T17:30:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: mon scrub interval = 300 2023-10-30T17:30:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on pool no app = False 2023-10-30T17:30:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: paxos service trim min = 10 2023-10-30T17:30:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-30T17:30:45.176 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2023-10-30T17:30:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2023-10-30T17:30:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2023-10-30T17:30:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2023-10-30T17:30:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T17:30:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T17:30:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2023-10-30T17:30:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2023-10-30T17:30:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: debug objecter = 20 2023-10-30T17:30:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: debug rados = 20 2023-10-30T17:30:45.177 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2023-10-30T17:30:45.178 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2023-10-30T17:30:45.178 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2023-10-30T17:30:45.178 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T17:30:45.179 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T17:30:45.242 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T17:30:45.243 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:30:45.243 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T17:30:45.257 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:30:45.258 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T17:30:45.312 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T17:30:45.312 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:30:45.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T17:30:45.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T17:30:45.390 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:30:45.391 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:30:45.444 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T17:30:45.445 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:30:45.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T17:30:45.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T17:30:45.484 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:30:45.485 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:30:45.540 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T17:30:45.540 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T17:30:45.541 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T17:30:45.595 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T17:30:45.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T17:30:45.677 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T17:30:45.680 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T17:30:45.680 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T17:30:45.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T17:30:45.724 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T17:30:45.739 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-30T17:30:45.780 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-30T17:30:45.796 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T17:30:45.796 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T17:30:45.796 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T17:30:45.823 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T17:30:45.823 INFO:tasks.ceph:role: osd.0 2023-10-30T17:30:45.823 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T17:30:45.823 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T17:30:45.889 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T17:30:45.890 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T17:30:45.890 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T17:30:45.890 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T17:30:45.890 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T17:30:45.890 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-30T17:30:45.891 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T17:30:45.891 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T17:30:45.891 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T17:30:45.891 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T17:30:45.921 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-30T17:30:45.923 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-30T17:30:45.924 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T17:30:45.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T17:30:46.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T17:30:46.131 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T17:30:46.132 INFO:tasks.ceph:role: osd.1 2023-10-30T17:30:46.132 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T17:30:46.132 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T17:30:46.198 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T17:30:46.198 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T17:30:46.198 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T17:30:46.199 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T17:30:46.199 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T17:30:46.199 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-30T17:30:46.199 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T17:30:46.199 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T17:30:46.200 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T17:30:46.200 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T17:30:46.231 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-30T17:30:46.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-30T17:30:46.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T17:30:46.303 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T17:30:46.376 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T17:30:46.442 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T17:30:46.443 INFO:tasks.ceph:role: osd.2 2023-10-30T17:30:46.443 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T17:30:46.443 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T17:30:46.509 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T17:30:46.510 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T17:30:46.510 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T17:30:46.510 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T17:30:46.510 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T17:30:46.511 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-30T17:30:46.511 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T17:30:46.511 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T17:30:46.511 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T17:30:46.511 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T17:30:46.542 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-30T17:30:46.544 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-30T17:30:46.545 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T17:30:46.617 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T17:30:46.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T17:30:46.756 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T17:30:46.756 INFO:tasks.ceph:role: osd.3 2023-10-30T17:30:46.756 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T17:30:46.757 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T17:30:46.822 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T17:30:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T17:30:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T17:30:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T17:30:46.823 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T17:30:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-30T17:30:46.823 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T17:30:46.824 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T17:30:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T17:30:46.824 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T17:30:46.854 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-30T17:30:46.856 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-30T17:30:46.857 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-30T17:30:46.928 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T17:30:46.999 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:30:47.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:47.102+0000 7fd965d94500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T17:30:47.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:47.102+0000 7fd965d94500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T17:30:47.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:47.102+0000 7fd965d94500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T17:30:47.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:47.102+0000 7fd965d94500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T17:30:47.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:47.102+0000 7fd965d94500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T17:30:47.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:47.103+0000 7fd965d94500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T17:30:49.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T17:30:49.969 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:30:50.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:50.056+0000 7ff980e9e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T17:30:50.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:50.056+0000 7ff980e9e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T17:30:50.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:50.056+0000 7ff980e9e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T17:30:50.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:50.056+0000 7ff980e9e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T17:30:50.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:50.056+0000 7ff980e9e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T17:30:50.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:50.057+0000 7ff980e9e500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T17:30:52.893 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T17:30:52.921 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:30:53.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:53.007+0000 7f891faa5500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T17:30:53.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:53.007+0000 7f891faa5500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T17:30:53.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:53.007+0000 7f891faa5500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T17:30:53.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:53.007+0000 7f891faa5500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T17:30:53.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:53.007+0000 7f891faa5500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T17:30:53.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:53.008+0000 7f891faa5500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T17:30:55.844 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T17:30:55.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:30:55.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:55.964+0000 7fa5ca907500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T17:30:55.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:55.965+0000 7fa5ca907500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T17:30:55.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:55.965+0000 7fa5ca907500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T17:30:55.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:55.965+0000 7fa5ca907500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T17:30:55.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:55.965+0000 7fa5ca907500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T17:30:55.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:30:55.965+0000 7fa5ca907500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T17:30:58.813 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T17:30:58.844 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T17:30:58.876 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T17:30:58.876 INFO:tasks.ceph:role: osd.4 2023-10-30T17:30:58.876 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T17:30:58.877 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T17:30:58.954 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T17:30:58.954 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T17:30:58.954 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T17:30:58.954 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T17:30:58.955 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T17:30:58.955 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2023-10-30T17:30:58.955 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T17:30:58.955 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T17:30:58.955 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T17:30:58.956 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T17:30:59.462 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2023-10-30T17:30:59.465 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2023-10-30T17:30:59.465 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-30T17:30:59.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T17:30:59.647 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T17:30:59.716 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T17:30:59.716 INFO:tasks.ceph:role: osd.5 2023-10-30T17:30:59.717 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T17:30:59.717 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T17:30:59.792 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T17:30:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T17:30:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T17:30:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T17:30:59.793 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T17:30:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2023-10-30T17:30:59.794 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T17:30:59.794 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T17:30:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T17:30:59.794 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T17:31:00.243 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2023-10-30T17:31:00.245 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2023-10-30T17:31:00.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-30T17:31:00.339 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T17:31:00.373 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-30T17:31:00.449 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T17:31:00.449 INFO:tasks.ceph:role: osd.6 2023-10-30T17:31:00.449 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T17:31:00.450 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T17:31:00.526 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T17:31:00.526 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T17:31:00.526 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T17:31:00.527 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T17:31:00.527 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T17:31:00.527 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2023-10-30T17:31:00.527 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T17:31:00.527 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T17:31:00.528 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T17:31:00.528 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T17:31:01.021 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2023-10-30T17:31:01.023 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2023-10-30T17:31:01.023 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-30T17:31:01.177 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-30T17:31:01.215 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-30T17:31:01.284 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T17:31:01.285 INFO:tasks.ceph:role: osd.7 2023-10-30T17:31:01.285 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T17:31:01.285 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T17:31:01.361 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T17:31:01.361 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T17:31:01.361 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T17:31:01.362 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T17:31:01.362 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T17:31:01.362 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2023-10-30T17:31:01.362 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T17:31:01.363 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T17:31:01.363 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T17:31:01.363 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T17:31:01.870 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2023-10-30T17:31:01.872 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2023-10-30T17:31:01.873 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-30T17:31:01.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-30T17:31:02.031 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:31:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:02.139+0000 7f75dce12500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T17:31:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:02.139+0000 7f75dce12500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T17:31:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:02.139+0000 7f75dce12500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T17:31:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:02.139+0000 7f75dce12500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T17:31:02.142 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:02.139+0000 7f75dce12500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T17:31:02.142 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:02.140+0000 7f75dce12500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T17:31:04.987 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T17:31:05.018 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:31:05.109 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:05.107+0000 7f0e01e0e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T17:31:05.109 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:05.107+0000 7f0e01e0e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T17:31:05.109 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:05.107+0000 7f0e01e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T17:31:05.109 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:05.107+0000 7f0e01e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T17:31:05.110 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:05.107+0000 7f0e01e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T17:31:05.110 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:05.108+0000 7f0e01e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T17:31:07.948 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T17:31:07.979 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:31:08.069 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:08.067+0000 7f882e929500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-30T17:31:08.069 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:08.067+0000 7f882e929500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-30T17:31:08.069 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:08.067+0000 7f882e929500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T17:31:08.070 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:08.067+0000 7f882e929500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T17:31:08.070 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:08.067+0000 7f882e929500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T17:31:08.070 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:08.068+0000 7f882e929500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-30T17:31:10.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-30T17:31:10.943 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:31:11.033 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:11.031+0000 7f2736a14500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-30T17:31:11.033 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:11.031+0000 7f2736a14500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-30T17:31:11.033 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:11.031+0000 7f2736a14500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T17:31:11.033 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:11.031+0000 7f2736a14500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T17:31:11.034 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:11.031+0000 7f2736a14500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T17:31:11.034 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:11.032+0000 7f2736a14500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-30T17:31:13.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-30T17:31:13.901 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T17:31:13.901 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:31:13.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-30T17:31:13.935 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:31:13.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T17:31:14.006 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:31:14.006 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T17:31:14.075 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:31:14.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T17:31:14.144 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:31:14.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T17:31:14.213 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:31:14.213 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T17:31:14.245 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:31:14.245 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T17:31:14.315 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:31:14.315 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T17:31:14.384 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:31:14.385 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-30T17:31:14.453 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:31:14.453 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-30T17:31:14.522 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:31:14.522 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T17:31:14.539 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:31:14.539 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-30T17:31:14.578 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T17:31:14.578 DEBUG:teuthology.orchestra.run.smithi170:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T17:31:14.581 DEBUG:teuthology.orchestra.run.smithi177:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T17:31:14.622 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.y] 2023-10-30T17:31:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBF6D9lzp55IxAA76elLww9b2vQcDeh3lZgLw== 2023-10-30T17:31:14.623 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2023-10-30T17:31:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBH6D9lnWAsBhAAdFjxPaWHg1zl/gLi+Ql48g== 2023-10-30T17:31:14.623 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2023-10-30T17:31:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBK6D9lBCFwAxAAHfFKH2kWuNwus+LPkhzUfg== 2023-10-30T17:31:14.623 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2023-10-30T17:31:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBN6D9lCEyCABAA73b1WHw9wlBp9CsAomykUA== 2023-10-30T17:31:14.624 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.3] 2023-10-30T17:31:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBP6D9lXWKQORAAmYsHWtQaeSAqIY4uWmY24g== 2023-10-30T17:31:14.624 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.x] 2023-10-30T17:31:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBF6D9l2K9aKBAAfOaUJUJCZBDkIGh+RmzoYA== 2023-10-30T17:31:14.625 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.4] 2023-10-30T17:31:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBW6D9lU5ddCBAAnjrffWLoOq0rj+VhTMu3LQ== 2023-10-30T17:31:14.625 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.5] 2023-10-30T17:31:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBZ6D9lACV4BhAAEMIFw8twAnHxWRaaq0X2cQ== 2023-10-30T17:31:14.625 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.6] 2023-10-30T17:31:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBc6D9l4sUWBBAA2teBMLSpsolMZCfbMdeHrQ== 2023-10-30T17:31:14.626 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.7] 2023-10-30T17:31:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBf6D9lwRDvARAA111Q35rt0bpQHLjdEQpemQ== 2023-10-30T17:31:14.626 INFO:teuthology.orchestra.run.smithi170.stdout:[client.0] 2023-10-30T17:31:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBF6D9lpRkoKxAAODACIbx38WQZmKEc1dQU2Q== 2023-10-30T17:31:14.626 INFO:teuthology.orchestra.run.smithi170.stdout:[client.1] 2023-10-30T17:31:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBF6D9l/9mGLhAAd9yoprOTTF6zoetHVgSgUA== 2023-10-30T17:31:14.645 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.y] 2023-10-30T17:31:14.645 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBF6D9lzp55IxAA76elLww9b2vQcDeh3lZgLw== 2023-10-30T17:31:14.646 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.0] 2023-10-30T17:31:14.646 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBH6D9lnWAsBhAAdFjxPaWHg1zl/gLi+Ql48g== 2023-10-30T17:31:14.646 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.1] 2023-10-30T17:31:14.646 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBK6D9lBCFwAxAAHfFKH2kWuNwus+LPkhzUfg== 2023-10-30T17:31:14.646 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.2] 2023-10-30T17:31:14.647 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBN6D9lCEyCABAA73b1WHw9wlBp9CsAomykUA== 2023-10-30T17:31:14.647 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.3] 2023-10-30T17:31:14.647 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBP6D9lXWKQORAAmYsHWtQaeSAqIY4uWmY24g== 2023-10-30T17:31:14.647 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.x] 2023-10-30T17:31:14.647 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBF6D9l2K9aKBAAfOaUJUJCZBDkIGh+RmzoYA== 2023-10-30T17:31:14.648 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.4] 2023-10-30T17:31:14.648 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBW6D9lU5ddCBAAnjrffWLoOq0rj+VhTMu3LQ== 2023-10-30T17:31:14.648 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.5] 2023-10-30T17:31:14.648 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBZ6D9lACV4BhAAEMIFw8twAnHxWRaaq0X2cQ== 2023-10-30T17:31:14.648 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.6] 2023-10-30T17:31:14.649 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBc6D9l4sUWBBAA2teBMLSpsolMZCfbMdeHrQ== 2023-10-30T17:31:14.649 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.7] 2023-10-30T17:31:14.649 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBf6D9lwRDvARAA111Q35rt0bpQHLjdEQpemQ== 2023-10-30T17:31:14.649 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2023-10-30T17:31:14.650 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBF6D9lpRkoKxAAODACIbx38WQZmKEc1dQU2Q== 2023-10-30T17:31:14.650 INFO:teuthology.orchestra.run.smithi177.stdout:[client.1] 2023-10-30T17:31:14.650 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBF6D9l/9mGLhAAd9yoprOTTF6zoetHVgSgUA== 2023-10-30T17:31:14.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T17:31:14.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T17:31:14.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:14.747 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:14.799 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:14.801 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:14.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:14.856 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:14.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:14.910 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:14.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T17:31:14.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T17:31:15.015 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:15.018 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:15.069 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:15.071 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:15.122 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:15.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:15.178 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:15.180 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T17:31:15.231 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T17:31:15.233 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T17:31:15.288 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T17:31:15.290 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T17:31:15.344 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T17:31:15.345 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T17:31:15.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T17:31:15.468 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T17:31:15.495 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T17:31:15.561 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T17:31:15.658 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T17:31:15.688 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T17:31:15.715 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T17:31:15.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T17:31:15.840 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:31:15.842 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T17:31:15.894 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T17:31:15.895 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T17:31:15.895 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T17:31:15.896 DEBUG:teuthology.orchestra.run.smithi170:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-10-30T17:31:15.898 INFO:tasks.ceph.mon.a:Started 2023-10-30T17:31:15.899 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T17:31:15.899 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T17:31:15.900 DEBUG:teuthology.orchestra.run.smithi170:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-10-30T17:31:15.902 INFO:tasks.ceph.mon.c:Started 2023-10-30T17:31:15.903 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-10-30T17:31:15.903 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T17:31:15.903 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-10-30T17:31:15.936 INFO:tasks.ceph.mon.b:Started 2023-10-30T17:31:15.936 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T17:31:15.937 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-30T17:31:15.937 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-10-30T17:31:15.939 INFO:tasks.ceph.mgr.y:Started 2023-10-30T17:31:15.940 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T17:31:15.940 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-10-30T17:31:15.942 INFO:tasks.ceph.mgr.x:Started 2023-10-30T17:31:15.943 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T17:31:15.943 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T17:31:16.045 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.043+0000 7f92aeafc640 10 monclient: get_monmap_and_config 2023-10-30T17:31:16.045 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.043+0000 7f92aeafc640 10 monclient: build_initial_monmap 2023-10-30T17:31:16.045 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.043+0000 7f92aeafc640 10 monclient: monmap: 2023-10-30T17:31:16.045 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:31:16.046 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:16.046 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:31:16.044691+0000 2023-10-30T17:31:16.046 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:31:16.044691+0000 2023-10-30T17:31:16.046 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:16.046 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:16.047 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:16.047 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:16.047 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:16.047 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:16.047 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:16.048 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:16.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.044+0000 7f92aeafc640 1 Processor -- start 2023-10-30T17:31:16.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.044+0000 7f92aeafc640 1 -- start start 2023-10-30T17:31:16.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.044+0000 7f92aeafc640 10 monclient: init 2023-10-30T17:31:16.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:16.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-10-30T17:31:16.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f92a80a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:16.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 10 monclient(hunting): picked mon.noname-b con 0x7f92a80b4500 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:16.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 0x7f92a813f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:16.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 10 monclient(hunting): picked mon.noname-a con 0x7f92a813ecc0 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:16.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92a8141ae0 addr v1:172.21.15.177:6789/0 2023-10-30T17:31:16.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a80a8b60 con 0x7f92a8141ae0 2023-10-30T17:31:16.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:16.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a80b48e0 con 0x7f92a813ecc0 2023-10-30T17:31:16.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:16.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a80aa5a0 con 0x7f92a80b4500 2023-10-30T17:31:16.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:16.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92aeafc640 10 monclient(hunting): authenticate will time out at 1258.368214s 2023-10-30T17:31:16.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92ac898640 1 -- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 msgr2=0x7f92a80a84b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3301/0 2023-10-30T17:31:16.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92ac898640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f92a80a84b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:16.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92a7fff640 1 -- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 msgr2=0x7f92a813f110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3300/0 2023-10-30T17:31:16.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92ad099640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f92a8141ae0 legacy=0x7f92a8141f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2023-10-30T17:31:16.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.045+0000 7f92a7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 0x7f92a813f110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:16.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.246+0000 7f92ac898640 1 -- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 msgr2=0x7f92a80a84b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3301/0 2023-10-30T17:31:16.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.246+0000 7f92a7fff640 1 -- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 msgr2=0x7f92a813f110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3300/0 2023-10-30T17:31:16.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.246+0000 7f92ac898640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f92a80a84b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:16.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.246+0000 7f92a7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 0x7f92a813f110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:16.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.246+0000 7f92ad099640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f92a8141ae0 legacy=0x7f92a8141f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2023-10-30T17:31:16.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.647+0000 7f92ac898640 1 -- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 msgr2=0x7f92a80a84b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3301/0 2023-10-30T17:31:16.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.647+0000 7f92a7fff640 1 -- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 msgr2=0x7f92a813f110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3300/0 2023-10-30T17:31:16.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.647+0000 7f92ac898640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f92a80a84b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:16.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.647+0000 7f92a7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 0x7f92a813f110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:16.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:16.647+0000 7f92ad099640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f92a8141ae0 legacy=0x7f92a8141f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2023-10-30T17:31:17.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:17.448+0000 7f92ac898640 1 -- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 msgr2=0x7f92a80a84b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3301/0 2023-10-30T17:31:17.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:17.448+0000 7f92a7fff640 1 -- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 msgr2=0x7f92a813f110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3300/0 2023-10-30T17:31:17.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:17.448+0000 7f92ac898640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f92a80a84b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:17.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:17.448+0000 7f92a7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 0x7f92a813f110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:17.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:17.448+0000 7f92ad099640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f92a8141ae0 legacy=0x7f92a8141f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2023-10-30T17:31:19.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 10 monclient(hunting): tick 2023-10-30T17:31:19.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T17:31:19.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 1 monclient(hunting): continuing hunt 2023-10-30T17:31:19.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T17:31:19.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 1 -- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 msgr2=0x7f92a80a84b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:19.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f92a80a84b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:19.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 1 -- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 msgr2=0x7f92a813f110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:19.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f92a813ecc0 0x7f92a813f110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:19.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f92a8141ae0 legacy=0x7f92a8141f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:19.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 10 monclient: _add_conns ranks=[4,0] 2023-10-30T17:31:19.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 0x7f928c005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:19.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-c con 0x7f928c0036c0 addr v2:172.21.15.177:3300/0 2023-10-30T17:31:19.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f928c006190 addr v1:172.21.15.170:6789/0 2023-10-30T17:31:19.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a80a8ff0 con 0x7f928c006190 2023-10-30T17:31:19.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:19.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a80b48e0 con 0x7f928c0036c0 2023-10-30T17:31:19.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a67fc640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:19.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92a7fff640 1 -- >> v1:172.21.15.170:6789/0 conn(0x7f928c006190 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.170:6789/0 2023-10-30T17:31:19.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92ac898640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2023-10-30T17:31:19.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.045+0000 7f92ac898640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:19.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.246+0000 7f92a7fff640 1 -- >> v1:172.21.15.170:6789/0 conn(0x7f928c006190 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.170:6789/0 2023-10-30T17:31:19.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.246+0000 7f92ac898640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2023-10-30T17:31:19.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.246+0000 7f92ac898640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:19.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.647+0000 7f92a7fff640 1 -- >> v1:172.21.15.170:6789/0 conn(0x7f928c006190 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.170:6789/0 2023-10-30T17:31:19.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.647+0000 7f92ac898640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2023-10-30T17:31:19.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:19.647+0000 7f92ac898640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:20.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:20.448+0000 7f92a7fff640 1 -- >> v1:172.21.15.170:6789/0 conn(0x7f928c006190 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.170:6789/0 2023-10-30T17:31:20.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:20.448+0000 7f92ac898640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2023-10-30T17:31:20.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:20.448+0000 7f92ac898640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:22.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 10 monclient(hunting): tick 2023-10-30T17:31:22.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T17:31:22.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 1 monclient(hunting): continuing hunt 2023-10-30T17:31:22.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T17:31:22.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:22.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f928c0036c0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:22.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 1 -- >> v1:172.21.15.170:6789/0 conn(0x7f928c006190 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:22.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-10-30T17:31:22.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92ad099640 1 -- reap_dead start 2023-10-30T17:31:22.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f928c009f00 addr v1:172.21.15.170:6789/0 2023-10-30T17:31:22.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f928c00a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:22.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-b con 0x7f92a80b4500 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:22.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 0x7f928c005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:22.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92ac898640 1 -- >> v1:172.21.15.170:6789/0 conn(0x7f928c009f00 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.170:6789/0 2023-10-30T17:31:22.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-a con 0x7f928c00a8f0 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:22.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a80a8b60 con 0x7f928c009f00 2023-10-30T17:31:22.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a7fff640 1 -- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 msgr2=0x7f928c00a380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3301/0 2023-10-30T17:31:22.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:22.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a80b48e0 con 0x7f928c00a8f0 2023-10-30T17:31:22.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:22.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a80aa5a0 con 0x7f92a80b4500 2023-10-30T17:31:22.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92ad099640 1 -- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3300/0 2023-10-30T17:31:22.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a67fc640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:22.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92a7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f928c00a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:22.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.045+0000 7f92ad099640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:22.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.246+0000 7f92a7fff640 1 -- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 msgr2=0x7f928c00a380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3301/0 2023-10-30T17:31:22.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.246+0000 7f92ac898640 1 -- >> v1:172.21.15.170:6789/0 conn(0x7f928c009f00 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.170:6789/0 2023-10-30T17:31:22.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.246+0000 7f92ad099640 1 -- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3300/0 2023-10-30T17:31:22.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.246+0000 7f92a7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f928c00a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:22.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.246+0000 7f92ad099640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:22.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.647+0000 7f92ac898640 1 -- >> v1:172.21.15.170:6789/0 conn(0x7f928c009f00 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.170:6789/0 2023-10-30T17:31:22.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.647+0000 7f92ad099640 1 -- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3300/0 2023-10-30T17:31:22.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.647+0000 7f92a7fff640 1 -- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 msgr2=0x7f928c00a380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3301/0 2023-10-30T17:31:22.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.647+0000 7f92ad099640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:22.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:22.647+0000 7f92a7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f928c00a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:23.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:23.448+0000 7f92ad099640 1 -- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3300/0 2023-10-30T17:31:23.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:23.448+0000 7f92ac898640 1 -- >> v1:172.21.15.170:6789/0 conn(0x7f928c009f00 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.170:6789/0 2023-10-30T17:31:23.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:23.448+0000 7f92a7fff640 1 -- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 msgr2=0x7f928c00a380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.170:3301/0 2023-10-30T17:31:23.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:23.448+0000 7f92ad099640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:23.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:23.448+0000 7f92a7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f928c00a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:23.603 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:00:07.542 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:31:23.914 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:00:07.853 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:31:23.945 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:00:07.911 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:31:23.946 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:00:07.911 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:31:24.271 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:00:08.236 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:31:24.271 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:00:08.236 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:31:25.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 10 monclient(hunting): tick 2023-10-30T17:31:25.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T17:31:25.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 1 monclient(hunting): continuing hunt 2023-10-30T17:31:25.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T17:31:25.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 1 -- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 msgr2=0x7f928c00a380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:25.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80b4500 0x7f928c00a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:25.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 1 -- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:25.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f928c00a8f0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:25.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 1 -- >> v1:172.21.15.170:6789/0 conn(0x7f928c009f00 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:25.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 10 monclient: _add_conns ranks=[1,4] 2023-10-30T17:31:25.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f928c00d0c0 addr v1:172.21.15.177:6789/0 2023-10-30T17:31:25.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:25.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-c con 0x7f928c011140 addr v2:172.21.15.177:3300/0 2023-10-30T17:31:25.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a80a8ff0 con 0x7f928c00d0c0 2023-10-30T17:31:25.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:25.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a80b48e0 con 0x7f928c011140 2023-10-30T17:31:25.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a67fc640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:25.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a7fff640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2023-10-30T17:31:25.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92ac898640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7f928c00d0c0 legacy=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2023-10-30T17:31:25.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.045+0000 7f92a7fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:25.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.441+0000 7f92a7fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:25.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.441+0000 7f92ac898640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f928c00d0c0 0x7f928c00d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:55452/0 (socket says 172.21.15.170:55452) 2023-10-30T17:31:25.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.441+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 learned_addr learned my addr 172.21.15.170:0/1381260190 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:31:25.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.519+0000 7f92a7fff640 10 monclient(hunting): get_auth_request con 0x7f928c011140 auth_method 0 2023-10-30T17:31:25.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.519+0000 7f92a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:25.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.519+0000 7f92a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:25.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.519+0000 7f92a7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:25.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.571+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v1:172.21.15.177:6789/0 conn(0x7f928c00d0c0 legacy=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:25.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.571+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v1:172.21.15.177:6789/0 conn(0x7f928c00d0c0 legacy=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:25.573 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.571+0000 7f92ac898640 1 --1- 172.21.15.170:0/1381260190 >> v1:172.21.15.177:6789/0 conn(0x7f928c00d0c0 0x7f928c00d4d0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2023-10-30T17:31:25.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.608+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T17:31:25.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.608+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:25.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.608+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:25.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:25.608+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:26.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:26.216+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:26.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:26.217+0000 7f92a7fff640 10 monclient(hunting): get_auth_request con 0x7f928c011140 auth_method 0 2023-10-30T17:31:26.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:26.217+0000 7f92a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:26.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:26.217+0000 7f92a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:26.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:26.217+0000 7f92a7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:26.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:26.218+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T17:31:26.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:26.218+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:26.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:26.218+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:26.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:26.218+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:27.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:27.020+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:27.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:27.020+0000 7f92a7fff640 10 monclient(hunting): get_auth_request con 0x7f928c011140 auth_method 0 2023-10-30T17:31:27.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:27.020+0000 7f92a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:27.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:27.020+0000 7f92a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:27.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:27.020+0000 7f92a7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:27.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:27.021+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T17:31:27.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:27.021+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:27.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:27.021+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:27.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:27.021+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:28.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 10 monclient(hunting): tick 2023-10-30T17:31:28.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T17:31:28.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 1 monclient(hunting): continuing hunt 2023-10-30T17:31:28.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T17:31:28.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:28.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c011140 0x7f928c013530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:28.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 >> v1:172.21.15.177:6789/0 conn(0x7f928c00d0c0 legacy=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:28.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-10-30T17:31:28.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 reap_dead start 2023-10-30T17:31:28.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f928c013a70 addr v1:172.21.15.177:6789/0 2023-10-30T17:31:28.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:28.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-b con 0x7f928c00a380 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:28.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:28.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-c con 0x7f928c008e20 addr v2:172.21.15.177:3300/0 2023-10-30T17:31:28.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a80a8b60 con 0x7f928c013a70 2023-10-30T17:31:28.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:28.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a80b48e0 con 0x7f928c008e20 2023-10-30T17:31:28.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:28.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a80aa5a0 con 0x7f928c00a380 2023-10-30T17:31:28.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a67fc640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:28.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:28.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a7fff640 10 monclient(hunting): get_auth_request con 0x7f928c00a380 auth_method 0 2023-10-30T17:31:28.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:28.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:28.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.046+0000 7f92a7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:28.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.047+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T17:31:28.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.047+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:28.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.047+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:28.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.047+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:28.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.052+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:28.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.052+0000 7f92ad099640 10 monclient(hunting): get_auth_request con 0x7f928c008e20 auth_method 0 2023-10-30T17:31:28.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.052+0000 7f92ad099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:28.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.052+0000 7f92ad099640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:28.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.052+0000 7f92ad099640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:28.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.053+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T17:31:28.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.053+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:28.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.053+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:28.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.053+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:28.249 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.248+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:28.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.248+0000 7f92a7fff640 10 monclient(hunting): get_auth_request con 0x7f928c00a380 auth_method 0 2023-10-30T17:31:28.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.248+0000 7f92a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:28.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.248+0000 7f92a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:28.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.248+0000 7f92a7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:28.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.249+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T17:31:28.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.249+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:28.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.249+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:28.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.249+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:28.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.254+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:28.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.254+0000 7f92ad099640 10 monclient(hunting): get_auth_request con 0x7f928c008e20 auth_method 0 2023-10-30T17:31:28.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.254+0000 7f92ad099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:28.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.254+0000 7f92ad099640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:28.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.254+0000 7f92ad099640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:28.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.255+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T17:31:28.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.255+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:28.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.255+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:28.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.255+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:28.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.650+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:28.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.651+0000 7f92a7fff640 10 monclient(hunting): get_auth_request con 0x7f928c00a380 auth_method 0 2023-10-30T17:31:28.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.651+0000 7f92a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:28.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.651+0000 7f92a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:28.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.651+0000 7f92a7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:28.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.651+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T17:31:28.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.651+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:28.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.652+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:28.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.652+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:28.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.657+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:28.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.657+0000 7f92ad099640 10 monclient(hunting): get_auth_request con 0x7f928c008e20 auth_method 0 2023-10-30T17:31:28.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.657+0000 7f92ad099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:28.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.657+0000 7f92ad099640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:28.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.657+0000 7f92ad099640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:28.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.658+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T17:31:28.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.658+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:28.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.658+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:28.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:28.658+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:29.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.453+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:29.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.453+0000 7f92a7fff640 10 monclient(hunting): get_auth_request con 0x7f928c00a380 auth_method 0 2023-10-30T17:31:29.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.453+0000 7f92a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:29.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.453+0000 7f92a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:29.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.453+0000 7f92a7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:29.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.454+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T17:31:29.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.454+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:29.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.454+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:29.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.454+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:29.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.459+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:29.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.460+0000 7f92ad099640 10 monclient(hunting): get_auth_request con 0x7f928c008e20 auth_method 0 2023-10-30T17:31:29.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.460+0000 7f92ad099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:29.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.460+0000 7f92ad099640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:29.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.460+0000 7f92ad099640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:29.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.461+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T17:31:29.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.461+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:29.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.461+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:29.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:29.461+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:31.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 10 monclient(hunting): tick 2023-10-30T17:31:31.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T17:31:31.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 1 monclient(hunting): continuing hunt 2023-10-30T17:31:31.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T17:31:31.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:31.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c00a380 0x7f928c00d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:31.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:31.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c008e20 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:31.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 >> v1:172.21.15.177:6789/0 conn(0x7f928c013a70 legacy=0x7f928c015e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:31.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 10 monclient: _add_conns ranks=[3,0] 2023-10-30T17:31:31.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:31.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-a con 0x7f928c009350 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:31.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f928c0098f0 addr v1:172.21.15.170:6789/0 2023-10-30T17:31:31.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f928c00da10 con 0x7f928c0098f0 2023-10-30T17:31:31.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:31.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a80b48e0 con 0x7f928c009350 2023-10-30T17:31:31.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92a67fc640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:31.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.046+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:31.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.047+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c009350 auth_method 0 2023-10-30T17:31:31.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.047+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:31.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.047+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:31.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.047+0000 7f92ac898640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:31.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.048+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:31.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.048+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:31.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.048+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:31.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.048+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:31.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.249+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:31.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.249+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c009350 auth_method 0 2023-10-30T17:31:31.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.249+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:31.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.249+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:31.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.249+0000 7f92ac898640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:31.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.250+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:31.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.250+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:31.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.250+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:31.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.250+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:31.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.651+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:31.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.652+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c009350 auth_method 0 2023-10-30T17:31:31.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.652+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:31.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.652+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:31.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.652+0000 7f92ac898640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.652+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.652+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.652+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:31.652+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:32.745 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:32.744+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:32.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:32.749+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c009350 auth_method 0 2023-10-30T17:31:32.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:32.749+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:32.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:32.749+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:32.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:32.749+0000 7f92ac898640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:32.759 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:32.758+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:32.759 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:32.758+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:32.759 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:32.758+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:32.760 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:32.758+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:34.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 10 monclient(hunting): tick 2023-10-30T17:31:34.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T17:31:34.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 1 monclient(hunting): continuing hunt 2023-10-30T17:31:34.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T17:31:34.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 msgr2=0x7f928c013530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:34.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c009350 0x7f928c013530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:34.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 >> v1:172.21.15.170:6789/0 conn(0x7f928c0098f0 legacy=0x7f928c008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:34.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 reap_dead start 2023-10-30T17:31:34.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-10-30T17:31:34.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:34.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-b con 0x7f928c008e20 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:34.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f928c009350 addr v1:172.21.15.177:6789/0 2023-10-30T17:31:34.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:34.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-a con 0x7f928c00a2e0 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:34.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f928c010180 con 0x7f928c009350 2023-10-30T17:31:34.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:34.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a80b48e0 con 0x7f928c00a2e0 2023-10-30T17:31:34.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:34.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a80aa5a0 con 0x7f928c008e20 2023-10-30T17:31:34.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.046+0000 7f92a67fc640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:34.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.047+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:34.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.047+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c008e20 auth_method 0 2023-10-30T17:31:34.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.047+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:34.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.047+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:34.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.047+0000 7f92ac898640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:34.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.047+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:34.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.048+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:34.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.048+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:34.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.048+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:34.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.048+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:34.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.048+0000 7f92ad099640 10 monclient(hunting): get_auth_request con 0x7f928c00a2e0 auth_method 0 2023-10-30T17:31:34.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.048+0000 7f92ad099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:34.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.048+0000 7f92ad099640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:34.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.048+0000 7f92ad099640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:34.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.049+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T17:31:34.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.049+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:34.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.049+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:34.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.049+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:34.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.249+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:34.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.249+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c008e20 auth_method 0 2023-10-30T17:31:34.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.249+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:34.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.249+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:34.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.249+0000 7f92ac898640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:34.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.250+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:34.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.250+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:34.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.250+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:34.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.250+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:34.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.250+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:34.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.250+0000 7f92ad099640 10 monclient(hunting): get_auth_request con 0x7f928c00a2e0 auth_method 0 2023-10-30T17:31:34.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.250+0000 7f92ad099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:34.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.250+0000 7f92ad099640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:34.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.250+0000 7f92ad099640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:34.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.252+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T17:31:34.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.252+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:34.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.252+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:34.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.252+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:34.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.651+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:34.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.651+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c008e20 auth_method 0 2023-10-30T17:31:34.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.651+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:34.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.651+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:34.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.651+0000 7f92ac898640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:34.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.652+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:34.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.652+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:34.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.652+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:34.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.652+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:34.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.653+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:34.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.653+0000 7f92ad099640 10 monclient(hunting): get_auth_request con 0x7f928c00a2e0 auth_method 0 2023-10-30T17:31:34.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.653+0000 7f92ad099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:34.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.653+0000 7f92ad099640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:34.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.653+0000 7f92ad099640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:34.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.654+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:34.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.654+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:34.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.654+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:34.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:34.654+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:35.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.454+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:35.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.454+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c008e20 auth_method 0 2023-10-30T17:31:35.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.454+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:35.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.454+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:35.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.454+0000 7f92ac898640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:35.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.455+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:35.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.455+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:35.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.455+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:35.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.455+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:35.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.456+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:35.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.456+0000 7f92ad099640 10 monclient(hunting): get_auth_request con 0x7f928c00a2e0 auth_method 0 2023-10-30T17:31:35.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.456+0000 7f92ad099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:35.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.456+0000 7f92ad099640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:35.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.456+0000 7f92ad099640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.457+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.457+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.457+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:35.457+0000 7f92ad099640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:37.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 10 monclient(hunting): tick 2023-10-30T17:31:37.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T17:31:37.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 1 monclient(hunting): continuing hunt 2023-10-30T17:31:37.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T17:31:37.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 msgr2=0x7f928c00d4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:37.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f928c008e20 0x7f928c00d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:37.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 msgr2=0x7f928c005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:37.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.170:3300/0 conn(0x7f928c00a2e0 0x7f928c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:37.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 >> v1:172.21.15.177:6789/0 conn(0x7f928c009350 legacy=0x7f928c010410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:37.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 10 monclient: _add_conns ranks=[4,0] 2023-10-30T17:31:37.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:37.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-c con 0x7f928c007650 addr v2:172.21.15.177:3300/0 2023-10-30T17:31:37.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f928c007fa0 addr v1:172.21.15.170:6789/0 2023-10-30T17:31:37.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f928c010b20 con 0x7f928c007fa0 2023-10-30T17:31:37.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:37.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 1 -- 172.21.15.170:0/1381260190 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a80b48e0 con 0x7f928c007650 2023-10-30T17:31:37.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.047+0000 7f92a67fc640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:37.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.048+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:37.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.049+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c007650 auth_method 0 2023-10-30T17:31:37.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.049+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:37.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.049+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:37.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.049+0000 7f92ac898640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:37.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.050+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 msgr2=0x7f928c007a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:37.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.050+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 msgr2=0x7f928c007a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:37.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.050+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:37.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.050+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T17:31:37.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.251+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:37.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.251+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c007650 auth_method 0 2023-10-30T17:31:37.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.251+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:37.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.251+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:37.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.251+0000 7f92ac898640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:37.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.252+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 msgr2=0x7f928c007a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:37.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.252+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 msgr2=0x7f928c007a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:37.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.252+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:37.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.252+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T17:31:37.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.654+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:37.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.654+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c007650 auth_method 0 2023-10-30T17:31:37.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.654+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:37.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.654+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:37.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.654+0000 7f92ac898640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:37.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.655+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 msgr2=0x7f928c007a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:37.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.656+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 msgr2=0x7f928c007a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:37.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.656+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:37.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:37.656+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T17:31:38.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:38.481+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:38.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:38.482+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f928c007650 auth_method 0 2023-10-30T17:31:38.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:38.482+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:38.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:38.482+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:38.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:38.482+0000 7f92ac898640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:38.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:38.482+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 msgr2=0x7f928c007a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T17:31:38.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:38.482+0000 7f92ac898640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 msgr2=0x7f928c007a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T17:31:38.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:38.482+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T17:31:38.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:38.482+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T17:31:39.404 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.402+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303351785 0 0) 0x7f928c010b20 con 0x7f928c007fa0 2023-10-30T17:31:39.404 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.402+0000 7f92a77fe640 10 monclient(hunting): _negotiate 2023-10-30T17:31:39.404 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.402+0000 7f92a77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:39.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.403+0000 7f92a77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:39.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.403+0000 7f92a77fe640 10 monclient(hunting): my global_id is 4098 2023-10-30T17:31:39.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.403+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9298003520 con 0x7f928c007fa0 2023-10-30T17:31:39.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1752672611 0 0) 0x7f9298003520 con 0x7f928c007fa0 2023-10-30T17:31:39.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 msgr2=0x7f928c007a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T17:31:39.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 --2- 172.21.15.170:0/1381260190 >> v2:172.21.15.177:3300/0 conn(0x7f928c007650 0x7f928c007a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:39.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient: _finish_hunting 0 2023-10-30T17:31:39.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 monclient: found mon.noname-a-legacy 2023-10-30T17:31:39.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:39.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.170:6789/0 2023-10-30T17:31:39.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928c008ac0 con 0x7f928c007fa0 2023-10-30T17:31:39.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient: _finish_auth 0 2023-10-30T17:31:39.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient: _check_auth_tickets 2023-10-30T17:31:39.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:39.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2465063691 0 0) 0x7f9294010610 con 0x7f928c007fa0 2023-10-30T17:31:39.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:39.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T17:31:39.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient: dump: 2023-10-30T17:31:39.454 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:31:39.454 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:39.454 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:39.454 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:39.455 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:39.455 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:39.455 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:39.455 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:39.455 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:39.455 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:39.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] but i'm connected to v1:172.21.15.170:6789/0, reconnecting 2023-10-30T17:31:39.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:39.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 >> v1:172.21.15.170:6789/0 conn(0x7f928c007fa0 legacy=0x7f928c0083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:39.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92ad099640 1 -- 172.21.15.170:0/1381260190 reap_dead start 2023-10-30T17:31:39.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:31:39.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 0x7f9298003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:39.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient(hunting): picked mon.c con 0x7f9298003970 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:39.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980047c0 0x7f9298004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:39.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient(hunting): picked mon.a con 0x7f92980047c0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:39.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92980051d0 0x7f929800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:39.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient(hunting): picked mon.b con 0x7f92980051d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:39.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:39.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a80b48e0 con 0x7f92980047c0 2023-10-30T17:31:39.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:39.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929800c100 con 0x7f92980051d0 2023-10-30T17:31:39.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:39.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.448+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929800c2c0 con 0x7f9298003970 2023-10-30T17:31:39.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.449+0000 7f92a77fe640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:39.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.449+0000 7f92a77fe640 10 monclient(hunting): _finish_auth 0 2023-10-30T17:31:39.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.449+0000 7f92ac898640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 0x7f9298003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:39.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.456+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f9298003970 auth_method 0 2023-10-30T17:31:39.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.456+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:39.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.456+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:39.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.456+0000 7f92ac898640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:39.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.481+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980047c0 0x7f9298004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:39.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.482+0000 7f92a7fff640 10 monclient(hunting): get_auth_request con 0x7f92980047c0 auth_method 0 2023-10-30T17:31:39.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.482+0000 7f92a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:39.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.482+0000 7f92a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:39.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.482+0000 7f92a7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:39.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.482+0000 7f92a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:39.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.482+0000 7f92a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:39.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.482+0000 7f92a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T17:31:39.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2023-10-30T17:31:39.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 msgr2=0x7f9298003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:39.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 0x7f9298003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:39.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92980051d0 msgr2=0x7f929800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:39.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92980051d0 0x7f929800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:39.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:39.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 1 monclient: found mon.a 2023-10-30T17:31:39.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:39.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:39.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 1 -- 172.21.15.170:0/1381260190 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9298003520 con 0x7f92980047c0 2023-10-30T17:31:39.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 10 monclient: _finish_auth 0 2023-10-30T17:31:39.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 10 monclient: _check_auth_tickets 2023-10-30T17:31:39.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:39.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92aeafc640 5 monclient: authenticate success, global_id 4098 2023-10-30T17:31:39.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.485+0000 7f92aeafc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:31:39.495 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.494+0000 7f92a7fff640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980047c0 0x7f9298004c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f928c009cd0 tx=0x7f92980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d9f72e187f4e9f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:39.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929402dcf0 con 0x7f92980047c0 2023-10-30T17:31:39.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:39.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:39.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92a77fe640 10 monclient: dump: 2023-10-30T17:31:39.529 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:31:39.530 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:39.530 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:39.530 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:39.530 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:39.531 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:39.531 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:39.531 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:39.531 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:39.531 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:39.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9294031070 con 0x7f92980047c0 2023-10-30T17:31:39.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92aeafc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:31:39.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92a77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:39.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92aeafc640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:39.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92a77fe640 1 -- 172.21.15.170:0/1381260190 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294037640 con 0x7f92980047c0 2023-10-30T17:31:39.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92aeafc640 4 set_mon_vals no callback set 2023-10-30T17:31:39.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:39.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:39.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.526+0000 7f92a77fe640 10 monclient: dump: 2023-10-30T17:31:39.534 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:31:39.534 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:39.535 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:39.535 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:39.535 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:39.535 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:39.535 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:39.536 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:39.536 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:39.536 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:39.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 10 monclient: shutdown 2023-10-30T17:31:39.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:39.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 1 -- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980047c0 msgr2=0x7f9298004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:39.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980047c0 0x7f9298004c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f928c009cd0 tx=0x7f92980057f0 comp rx=0 tx=0).stop 2023-10-30T17:31:39.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 1 -- 172.21.15.170:0/1381260190 shutdown_connections 2023-10-30T17:31:39.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92980051d0 0x7f929800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:39.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980047c0 0x7f9298004c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:39.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 1 --2- 172.21.15.170:0/1381260190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 0x7f9298003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:39.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 1 -- 172.21.15.170:0/1381260190 >> 172.21.15.170:0/1381260190 conn(0x7f92a8071b60 msgr2=0x7f92a8071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:39.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 1 -- 172.21.15.170:0/1381260190 shutdown_connections 2023-10-30T17:31:39.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.527+0000 7f92aeafc640 1 -- 172.21.15.170:0/1381260190 wait complete. 2023-10-30T17:31:39.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.528+0000 7f92aeafc640 10 monclient: build_initial_monmap 2023-10-30T17:31:39.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.528+0000 7f92aeafc640 10 monclient: monmap: 2023-10-30T17:31:39.542 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:31:39.542 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:39.542 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:31:39.542 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:31:39.542 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:39.543 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:39.543 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:39.543 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:39.543 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:39.543 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:39.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.528+0000 7f92aeafc640 1 librados: starting msgr at 2023-10-30T17:31:39.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.528+0000 7f92aeafc640 1 librados: starting objecter 2023-10-30T17:31:39.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.528+0000 7f92aeafc640 1 Processor -- start 2023-10-30T17:31:39.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.528+0000 7f92aeafc640 1 -- start start 2023-10-30T17:31:39.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.528+0000 7f92aeafc640 1 librados: setting wanted keys 2023-10-30T17:31:39.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.528+0000 7f92aeafc640 1 librados: calling monclient init 2023-10-30T17:31:39.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.528+0000 7f92aeafc640 10 monclient: init 2023-10-30T17:31:39.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:39.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:31:39.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 0x7f92a813f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:39.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 10 monclient(hunting): picked mon.noname-c con 0x7f9298003970 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:39.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92980047c0 0x7f92a81d7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:39.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 10 monclient(hunting): picked mon.noname-b con 0x7f92980047c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:39.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980051d0 0x7f92a8138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:39.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 10 monclient(hunting): picked mon.noname-a con 0x7f92980051d0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:39.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:39.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a81d6890 con 0x7f92980051d0 2023-10-30T17:31:39.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:39.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a81d6ae0 con 0x7f92980047c0 2023-10-30T17:31:39.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:39.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a80b55d0 con 0x7f9298003970 2023-10-30T17:31:39.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:39.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92aeafc640 10 monclient(hunting): authenticate will time out at 1281.852344s 2023-10-30T17:31:39.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.529+0000 7f92ac898640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 0x7f92a813f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:39.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92a7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92980047c0 0x7f92a81d7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:39.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92ac898640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 0x7f92a813f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:33006/0 (socket says 172.21.15.170:33006) 2023-10-30T17:31:39.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92ac898640 1 -- 172.21.15.170:0/2457832608 learned_addr learned my addr 172.21.15.170:0/2457832608 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:31:39.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92ac898640 10 monclient(hunting): get_auth_request con 0x7f9298003970 auth_method 0 2023-10-30T17:31:39.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92ac898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:39.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92ac898640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:39.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92ac898640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:39.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92a7fff640 10 monclient(hunting): get_auth_request con 0x7f92980047c0 auth_method 0 2023-10-30T17:31:39.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:39.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:39.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.530+0000 7f92a7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:39.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.532+0000 7f92a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:39.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.532+0000 7f92a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:39.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.532+0000 7f92a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:39.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.532+0000 7f92ac898640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:39.551 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.532+0000 7f92ac898640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:39.551 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.532+0000 7f92ac898640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:39.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.541+0000 7f92ad099640 1 --2- 172.21.15.170:0/2457832608 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980051d0 0x7f92a8138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:39.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.542+0000 7f92ad099640 10 monclient(hunting): get_auth_request con 0x7f92980051d0 auth_method 0 2023-10-30T17:31:39.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.542+0000 7f92ad099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:39.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.542+0000 7f92ad099640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:39.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.542+0000 7f92ad099640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:39.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.543+0000 7f92a7fff640 10 monclient(hunting): handle_auth_done global_id 4099 payload 824 2023-10-30T17:31:39.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.543+0000 7f92a7fff640 1 -- 172.21.15.170:0/2457832608 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 msgr2=0x7f92a813f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:39.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.543+0000 7f92a7fff640 1 --2- 172.21.15.170:0/2457832608 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 0x7f92a813f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:39.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.543+0000 7f92a7fff640 1 -- 172.21.15.170:0/2457832608 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980051d0 msgr2=0x7f92a8138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:39.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.543+0000 7f92a7fff640 1 --2- 172.21.15.170:0/2457832608 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980051d0 0x7f92a8138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:39.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.543+0000 7f92a7fff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:39.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.543+0000 7f92a7fff640 1 monclient: found mon.noname-b 2023-10-30T17:31:39.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.543+0000 7f92a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:39.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.543+0000 7f92a7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:31:39.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.543+0000 7f92a7fff640 1 -- 172.21.15.170:0/2457832608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a80b5790 con 0x7f92980047c0 2023-10-30T17:31:39.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.553+0000 7f92a7fff640 1 --2- 172.21.15.170:0/2457832608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92980047c0 0x7f92a81d7000 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f929800c0d0 tx=0x7f9294005a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:39.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.553+0000 7f92a57fa640 10 client.?.objecter ms_handle_connect 0x7f92980047c0 2023-10-30T17:31:39.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.553+0000 7f92a57fa640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:39.589 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 1 -- 172.21.15.170:0/2457832608 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294003dc0 con 0x7f92980047c0 2023-10-30T17:31:39.590 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:39.590 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:31:39.590 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 10 monclient: dump: 2023-10-30T17:31:39.590 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:31:39.590 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:39.591 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:39.591 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:39.591 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:39.591 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:39.591 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:39.591 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:39.592 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:39.592 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:39.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 10 monclient: _finish_auth 0 2023-10-30T17:31:39.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 10 monclient: _check_auth_tickets 2023-10-30T17:31:39.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:39.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 1 -- 172.21.15.170:0/2457832608 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929400c040 con 0x7f92980047c0 2023-10-30T17:31:39.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:39.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 1 -- 172.21.15.170:0/2457832608 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294004240 con 0x7f92980047c0 2023-10-30T17:31:39.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92aeafc640 5 monclient: authenticate success, global_id 4099 2023-10-30T17:31:39.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92aeafc640 10 monclient: _renew_subs 2023-10-30T17:31:39.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92aeafc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:39.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92aeafc640 1 -- 172.21.15.170:0/2457832608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a81d7540 con 0x7f92980047c0 2023-10-30T17:31:39.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:39.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:31:39.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a57fa640 10 monclient: dump: 2023-10-30T17:31:39.595 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:31:39.595 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:39.595 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:39.595 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:39.595 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:39.596 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:39.596 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:39.596 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:39.596 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:39.596 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:39.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92a6ffd640 4 set_mon_vals no callback set 2023-10-30T17:31:39.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92aeafc640 10 client.4099.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:39.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92aeafc640 10 monclient: _renew_subs 2023-10-30T17:31:39.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92aeafc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:39.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92aeafc640 1 -- 172.21.15.170:0/2457832608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a8138c20 con 0x7f92980047c0 2023-10-30T17:31:39.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.588+0000 7f92aeafc640 1 librados: init done 2023-10-30T17:31:39.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.589+0000 7f92aeafc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:39.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.589+0000 7f92aeafc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:39.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.589+0000 7f92aeafc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:39.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.589+0000 7f92aeafc640 1 -- 172.21.15.170:0/2457832608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a8138c20 con 0x7f92980047c0 2023-10-30T17:31:39.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.810+0000 7f92a57fa640 1 -- 172.21.15.170:0/2457832608 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f92940043a0 con 0x7f92980047c0 2023-10-30T17:31:39.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.817+0000 7f92a57fa640 1 -- 172.21.15.170:0/2457832608 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9294006340 con 0x7f92980047c0 2023-10-30T17:31:39.818 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.817+0000 7f92a57fa640 10 client.4099.objecter ms_dispatch 0x7f92a8000b90 osd_map(1..1 src has 1..1) v4 2023-10-30T17:31:39.818 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.817+0000 7f92a57fa640 3 client.4099.objecter handle_osd_map got epochs [1,1] > 0 2023-10-30T17:31:39.818 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.817+0000 7f92a57fa640 3 client.4099.objecter handle_osd_map decoding full epoch 1 2023-10-30T17:31:39.818 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.817+0000 7f92a57fa640 20 client.4099.objecter dump_active .. 0 homeless 2023-10-30T17:31:39.819 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.817+0000 7f92a57fa640 1 -- 172.21.15.170:0/2457832608 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f9294044020 con 0x7f92980047c0 2023-10-30T17:31:39.819 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.817+0000 7f92a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:39.819 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.817+0000 7f92a57fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:39.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.879+0000 7f92aeafc640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T17:31:39.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.879+0000 7f92aeafc640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T17:31:39.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.879+0000 7f92aeafc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:39.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:39.879+0000 7f92aeafc640 1 -- 172.21.15.170:0/2457832608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f92a8138c20 con 0x7f92980047c0 2023-10-30T17:31:40.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.119+0000 7f92a57fa640 1 -- 172.21.15.170:0/2457832608 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9294033070 con 0x7f92980047c0 2023-10-30T17:31:40.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.119+0000 7f92a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T17:31:40.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.119+0000 7f92a57fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-10-30T17:31:40.121 INFO:teuthology.orchestra.run.smithi170.stderr:adjusted tunables profile to default 2023-10-30T17:31:40.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.119+0000 7f92aeafc640 10 librados: watch_flush enter 2023-10-30T17:31:40.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.119+0000 7f92aeafc640 10 librados: watch_flush exit 2023-10-30T17:31:40.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.119+0000 7f92aeafc640 20 client.4099.objecter shutdown clearing up homeless session... 2023-10-30T17:31:40.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.119+0000 7f92aeafc640 10 client.4099.objecter successfully canceled tick 2023-10-30T17:31:40.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 10 monclient: shutdown 2023-10-30T17:31:40.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:40.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 1 -- 172.21.15.170:0/2457832608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92980047c0 msgr2=0x7f92a81d7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:40.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 1 --2- 172.21.15.170:0/2457832608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92980047c0 0x7f92a81d7000 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f929800c0d0 tx=0x7f9294005a80 comp rx=0 tx=0).stop 2023-10-30T17:31:40.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 1 -- 172.21.15.170:0/2457832608 shutdown_connections 2023-10-30T17:31:40.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 1 --2- 172.21.15.170:0/2457832608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92980047c0 0x7f92a81d7000 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:40.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 1 --2- 172.21.15.170:0/2457832608 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92980051d0 0x7f92a8138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:40.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 1 --2- 172.21.15.170:0/2457832608 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9298003970 0x7f92a813f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:40.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 1 -- 172.21.15.170:0/2457832608 >> 172.21.15.170:0/2457832608 conn(0x7f92a8071b60 msgr2=0x7f92a81cc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:40.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 1 -- 172.21.15.170:0/2457832608 shutdown_connections 2023-10-30T17:31:40.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 1 -- 172.21.15.170:0/2457832608 wait complete. 2023-10-30T17:31:40.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:40.120+0000 7f92aeafc640 1 librados: shutdown 2023-10-30T17:31:40.131 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T17:31:40.132 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T17:31:40.132 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:31:40.133 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T17:31:40.160 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:31:40.160 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T17:31:40.227 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:31:40.227 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T17:31:40.294 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:31:40.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T17:31:40.362 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:31:40.362 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T17:31:40.389 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:31:40.389 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T17:31:40.455 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:31:40.456 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-30T17:31:40.522 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-30T17:31:40.522 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-30T17:31:40.529 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:40.528+0000 7f2fb4c2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T17:31:40.581 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:40.580+0000 7f2fb4c2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T17:31:40.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 26bebf29-e72f-410f-aff7-561160c246e5 0 2023-10-30T17:31:40.627 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:40.625+0000 7f2b9a22b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T17:31:40.682 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:40.681+0000 7f2b9a22b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T17:31:40.726 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.725+0000 7f132987c640 10 monclient: get_monmap_and_config 2023-10-30T17:31:40.727 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.725+0000 7f132987c640 10 monclient: build_initial_monmap 2023-10-30T17:31:40.727 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.725+0000 7f132987c640 10 monclient: monmap: 2023-10-30T17:31:40.727 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:40.727 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:40.728 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:31:40.726179+0000 2023-10-30T17:31:40.728 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:31:40.726179+0000 2023-10-30T17:31:40.728 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:40.728 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:40.728 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:40.729 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:40.729 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:40.729 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:40.729 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:40.729 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:40.732 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.725+0000 7f132987c640 1 Processor -- start 2023-10-30T17:31:40.732 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 1 -- start start 2023-10-30T17:31:40.732 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 10 monclient: init 2023-10-30T17:31:40.732 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:40.733 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-10-30T17:31:40.733 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13240b4500 addr v1:172.21.15.177:6789/0 2023-10-30T17:31:40.733 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f132413f3d0 0x7f13240a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:40.733 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 10 monclient(hunting): picked mon.noname-c con 0x7f132413f3d0 addr v2:172.21.15.177:3300/0 2023-10-30T17:31:40.733 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1324141ae0 0x7f1324141f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:40.734 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 10 monclient(hunting): picked mon.noname-b con 0x7f1324141ae0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:40.734 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13240a8b60 con 0x7f13240b4500 2023-10-30T17:31:40.734 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:40.734 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13240b48e0 con 0x7f132413f3d0 2023-10-30T17:31:40.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:40.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13240aa5a0 con 0x7f1324141ae0 2023-10-30T17:31:40.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:40.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.726+0000 7f132987c640 10 monclient(hunting): authenticate will time out at 1274.087671s 2023-10-30T17:31:40.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f131bfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f132413f3d0 0x7f13240a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:40.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f13237fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1324141ae0 0x7f1324141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:40.736 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f131bfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f132413f3d0 0x7f13240a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:43004/0 (socket says 172.21.15.177:43004) 2023-10-30T17:31:40.736 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f131bfff640 1 -- 172.21.15.177:0/3951021556 learned_addr learned my addr 172.21.15.177:0/3951021556 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:40.736 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f131bfff640 10 monclient(hunting): get_auth_request con 0x7f132413f3d0 auth_method 0 2023-10-30T17:31:40.736 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f131bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:40.736 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f131bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:40.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f131bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:40.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f13237fe640 10 monclient(hunting): get_auth_request con 0x7f1324141ae0 auth_method 0 2023-10-30T17:31:40.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f13237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:40.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f13237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:40.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.727+0000 7f13237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:40.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.728+0000 7f131bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:40.738 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.728+0000 7f131bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:40.738 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.728+0000 7f131bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:40.738 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.728+0000 7f13237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:40.738 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.728+0000 7f13237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:40.738 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.728+0000 7f13237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:40.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.730+0000 7f131bfff640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2023-10-30T17:31:40.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.730+0000 7f131bfff640 1 -- 172.21.15.177:0/3951021556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1324141ae0 msgr2=0x7f1324141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:40.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.730+0000 7f131bfff640 1 --2- 172.21.15.177:0/3951021556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1324141ae0 0x7f1324141f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:40.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.730+0000 7f131bfff640 1 -- 172.21.15.177:0/3951021556 >> v1:172.21.15.177:6789/0 conn(0x7f13240b4500 legacy=0x7f132413ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:40.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.730+0000 7f131bfff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:40.740 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.730+0000 7f131bfff640 1 monclient: found mon.noname-c 2023-10-30T17:31:40.740 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.730+0000 7f131bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:40.740 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.730+0000 7f131bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:31:40.740 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.730+0000 7f131bfff640 1 -- 172.21.15.177:0/3951021556 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13240a8ff0 con 0x7f132413f3d0 2023-10-30T17:31:40.741 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.735+0000 7f131bfff640 1 --2- 172.21.15.177:0/3951021556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f132413f3d0 0x7f13240a84b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f130c009870 tx=0x7f130c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b9944f16895e045 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:40.741 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 1 -- 172.21.15.177:0/3951021556 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c031640 con 0x7f132413f3d0 2023-10-30T17:31:40.741 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:40.742 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:31:40.742 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 10 monclient: dump: 2023-10-30T17:31:40.742 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:40.742 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:40.742 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:40.743 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:40.743 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:40.743 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:40.743 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:40.743 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:40.744 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:40.744 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:40.744 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 10 monclient: _finish_auth 0 2023-10-30T17:31:40.744 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 10 monclient: _check_auth_tickets 2023-10-30T17:31:40.744 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:40.745 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 1 -- 172.21.15.177:0/3951021556 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c0317a0 con 0x7f132413f3d0 2023-10-30T17:31:40.745 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:40.745 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 1 -- 172.21.15.177:0/3951021556 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c031b30 con 0x7f132413f3d0 2023-10-30T17:31:40.745 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f132987c640 5 monclient: authenticate success, global_id 4114 2023-10-30T17:31:40.745 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f132987c640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:40.746 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f132987c640 4 set_mon_vals no callback set 2023-10-30T17:31:40.746 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:40.746 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:31:40.746 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.739+0000 7f13227fc640 10 monclient: dump: 2023-10-30T17:31:40.746 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:40.747 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:40.747 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:40.747 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:40.747 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:40.747 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:40.747 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:40.748 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:40.748 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:40.748 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:40.752 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.740+0000 7f132987c640 10 monclient: shutdown 2023-10-30T17:31:40.752 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.740+0000 7f132987c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:40.752 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.740+0000 7f132987c640 1 -- 172.21.15.177:0/3951021556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f132413f3d0 msgr2=0x7f13240a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:40.752 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.740+0000 7f132987c640 1 --2- 172.21.15.177:0/3951021556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f132413f3d0 0x7f13240a84b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f130c009870 tx=0x7f130c02b8f0 comp rx=0 tx=0).stop 2023-10-30T17:31:40.753 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.740+0000 7f132987c640 1 -- 172.21.15.177:0/3951021556 shutdown_connections 2023-10-30T17:31:40.753 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.740+0000 7f132987c640 1 --2- 172.21.15.177:0/3951021556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1324141ae0 0x7f1324141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:40.753 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.740+0000 7f132987c640 1 --2- 172.21.15.177:0/3951021556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f132413f3d0 0x7f13240a84b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:40.753 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.740+0000 7f132987c640 1 -- 172.21.15.177:0/3951021556 >> 172.21.15.177:0/3951021556 conn(0x7f1324071b60 msgr2=0x7f1324071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:40.753 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.740+0000 7f132987c640 1 -- 172.21.15.177:0/3951021556 shutdown_connections 2023-10-30T17:31:40.754 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.740+0000 7f132987c640 1 -- 172.21.15.177:0/3951021556 wait complete. 2023-10-30T17:31:40.754 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.741+0000 7f132987c640 10 monclient: build_initial_monmap 2023-10-30T17:31:40.754 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.741+0000 7f132987c640 10 monclient: monmap: 2023-10-30T17:31:40.754 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:40.754 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:40.755 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2023-10-30T17:31:40.755 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2023-10-30T17:31:40.755 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:40.755 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:40.755 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:40.756 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:40.756 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:40.756 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:40.756 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.741+0000 7f132987c640 1 librados: starting msgr at 2023-10-30T17:31:40.756 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.741+0000 7f132987c640 1 librados: starting objecter 2023-10-30T17:31:40.757 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.741+0000 7f132987c640 1 Processor -- start 2023-10-30T17:31:40.757 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.741+0000 7f132987c640 1 -- start start 2023-10-30T17:31:40.757 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.741+0000 7f132987c640 1 librados: setting wanted keys 2023-10-30T17:31:40.757 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.741+0000 7f132987c640 1 librados: calling monclient init 2023-10-30T17:31:40.757 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.741+0000 7f132987c640 10 monclient: init 2023-10-30T17:31:40.758 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:40.758 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:31:40.758 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13240b4500 0x7f132413b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:40.758 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 10 monclient(hunting): picked mon.noname-b con 0x7f13240b4500 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:40.758 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132413f3d0 0x7f13241cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:40.759 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 10 monclient(hunting): picked mon.noname-a con 0x7f132413f3d0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:40.759 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1324141ae0 0x7f13240ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:40.759 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 10 monclient(hunting): picked mon.noname-c con 0x7f1324141ae0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:40.759 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:40.759 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13240ac470 con 0x7f132413f3d0 2023-10-30T17:31:40.760 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:40.760 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13240abf50 con 0x7f13240b4500 2023-10-30T17:31:40.760 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:40.760 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13240aaec0 con 0x7f1324141ae0 2023-10-30T17:31:40.760 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:40.761 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.742+0000 7f132987c640 10 monclient(hunting): authenticate will time out at 1274.103543s 2023-10-30T17:31:40.761 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f1322ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13240b4500 0x7f132413b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:40.761 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f13237fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1324141ae0 0x7f13240ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:40.761 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f1322ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13240b4500 0x7f132413b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:43012/0 (socket says 172.21.15.177:43012) 2023-10-30T17:31:40.761 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f1322ffd640 1 -- 172.21.15.177:0/2681568334 learned_addr learned my addr 172.21.15.177:0/2681568334 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:40.762 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f1322ffd640 10 monclient(hunting): get_auth_request con 0x7f13240b4500 auth_method 0 2023-10-30T17:31:40.762 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f1322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:40.762 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f1322ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:40.762 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f1322ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:40.763 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f13237fe640 10 monclient(hunting): get_auth_request con 0x7f1324141ae0 auth_method 0 2023-10-30T17:31:40.763 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f13237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:40.763 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f13237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:40.763 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.743+0000 7f13237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:40.763 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f1322ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:40.764 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f1322ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:40.764 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f1322ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:40.764 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f13237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:40.764 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f13237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:40.764 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f13237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:40.765 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f131bfff640 1 --2- 172.21.15.177:0/2681568334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132413f3d0 0x7f13241cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:40.765 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f131bfff640 10 monclient(hunting): get_auth_request con 0x7f132413f3d0 auth_method 0 2023-10-30T17:31:40.765 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f131bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:40.765 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f131bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:40.766 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.744+0000 7f131bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:40.766 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.745+0000 7f131bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:40.766 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.745+0000 7f131bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:40.766 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.745+0000 7f131bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:40.766 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.748+0000 7f131bfff640 10 monclient(hunting): handle_auth_done global_id 4113 payload 824 2023-10-30T17:31:40.766 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.748+0000 7f131bfff640 1 -- 172.21.15.177:0/2681568334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1324141ae0 msgr2=0x7f13240ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:40.767 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.748+0000 7f131bfff640 1 --2- 172.21.15.177:0/2681568334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1324141ae0 0x7f13240ac6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:40.767 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.748+0000 7f131bfff640 1 -- 172.21.15.177:0/2681568334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13240b4500 msgr2=0x7f132413b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:40.767 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.748+0000 7f131bfff640 1 --2- 172.21.15.177:0/2681568334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13240b4500 0x7f132413b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:40.767 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.748+0000 7f131bfff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:40.767 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.748+0000 7f131bfff640 1 monclient: found mon.noname-a 2023-10-30T17:31:40.768 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.748+0000 7f131bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:40.768 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.748+0000 7f131bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:40.768 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.748+0000 7f131bfff640 1 -- 172.21.15.177:0/2681568334 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c0094d0 con 0x7f132413f3d0 2023-10-30T17:31:40.768 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.750+0000 7f131bfff640 1 --2- 172.21.15.177:0/2681568334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132413f3d0 0x7f13241cc840 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f130c033000 tx=0x7f130c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:40.768 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.750+0000 7f1320ff9640 10 client.?.objecter ms_handle_connect 0x7f132413f3d0 2023-10-30T17:31:40.769 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.750+0000 7f1320ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:40.770 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 1 -- 172.21.15.177:0/2681568334 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c03ac60 con 0x7f132413f3d0 2023-10-30T17:31:40.770 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:40.771 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:40.771 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 10 monclient: dump: 2023-10-30T17:31:40.771 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:40.771 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:40.771 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:40.772 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:40.772 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:40.772 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:40.772 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:40.773 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:40.773 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:40.773 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:40.773 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 10 monclient: _finish_auth 0 2023-10-30T17:31:40.773 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 10 monclient: _check_auth_tickets 2023-10-30T17:31:40.774 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:40.774 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 1 -- 172.21.15.177:0/2681568334 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c03ade0 con 0x7f132413f3d0 2023-10-30T17:31:40.774 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f132987c640 5 monclient: authenticate success, global_id 4113 2023-10-30T17:31:40.774 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f132987c640 10 monclient: _renew_subs 2023-10-30T17:31:40.774 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f132987c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:40.775 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f132987c640 1 -- 172.21.15.177:0/2681568334 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13240abb00 con 0x7f132413f3d0 2023-10-30T17:31:40.775 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:40.775 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 1 -- 172.21.15.177:0/2681568334 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c044630 con 0x7f132413f3d0 2023-10-30T17:31:40.775 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:40.775 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:40.776 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1320ff9640 10 monclient: dump: 2023-10-30T17:31:40.776 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:40.776 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:40.776 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:40.776 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:40.777 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:40.777 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:40.777 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:40.777 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:40.777 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:40.778 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:40.778 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f1321ffb640 4 set_mon_vals no callback set 2023-10-30T17:31:40.778 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f132987c640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:40.778 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f132987c640 10 monclient: _renew_subs 2023-10-30T17:31:40.778 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f132987c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:40.779 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f132987c640 1 -- 172.21.15.177:0/2681568334 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13240acdc0 con 0x7f132413f3d0 2023-10-30T17:31:40.779 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.754+0000 7f132987c640 1 librados: init done 2023-10-30T17:31:40.779 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.755+0000 7f132987c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:40.779 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.755+0000 7f132987c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:40.779 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.755+0000 7f132987c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:40.780 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.755+0000 7f132987c640 1 -- 172.21.15.177:0/2681568334 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13240abd40 con 0x7f132413f3d0 2023-10-30T17:31:40.780 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.770+0000 7f1320ff9640 1 -- 172.21.15.177:0/2681568334 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f130c02f050 con 0x7f132413f3d0 2023-10-30T17:31:40.781 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.770+0000 7f1320ff9640 1 -- 172.21.15.177:0/2681568334 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f130c04d490 con 0x7f132413f3d0 2023-10-30T17:31:40.781 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.770+0000 7f1320ff9640 10 client.4113.objecter ms_dispatch 0x7f1324000b90 osd_map(2..2 src has 1..2) v4 2023-10-30T17:31:40.781 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.770+0000 7f1320ff9640 3 client.4113.objecter handle_osd_map got epochs [2,2] > 0 2023-10-30T17:31:40.781 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.770+0000 7f1320ff9640 3 client.4113.objecter handle_osd_map decoding full epoch 2 2023-10-30T17:31:40.781 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.770+0000 7f1320ff9640 20 client.4113.objecter dump_active .. 0 homeless 2023-10-30T17:31:40.886 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.885+0000 7f1320ff9640 1 -- 172.21.15.177:0/2681568334 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f130c035020 con 0x7f132413f3d0 2023-10-30T17:31:40.886 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.885+0000 7f1320ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:40.887 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.885+0000 7f1320ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:40.956 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.955+0000 7f132987c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "26bebf29-e72f-410f-aff7-561160c246e5", "id": 0}] 2023-10-30T17:31:40.956 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.955+0000 7f132987c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "26bebf29-e72f-410f-aff7-561160c246e5", "id": 0}] 2023-10-30T17:31:40.957 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.955+0000 7f132987c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:40.957 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:40.955+0000 7f132987c640 1 -- 172.21.15.177:0/2681568334 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "26bebf29-e72f-410f-aff7-561160c246e5", "id": 0} v 0) v1 -- 0x7f1324067cd0 con 0x7f132413f3d0 2023-10-30T17:31:41.076 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.074+0000 7f1320ff9640 1 -- 172.21.15.177:0/2681568334 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26bebf29-e72f-410f-aff7-561160c246e5", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f130c02d040 con 0x7f132413f3d0 2023-10-30T17:31:41.076 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.074+0000 7f1320ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "26bebf29-e72f-410f-aff7-561160c246e5", "id": 0}] 2023-10-30T17:31:41.076 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.074+0000 7f1320ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:41.077 INFO:teuthology.orchestra.run.smithi177.stdout:0 2023-10-30T17:31:41.078 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 10 librados: watch_flush enter 2023-10-30T17:31:41.078 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 10 librados: watch_flush exit 2023-10-30T17:31:41.078 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 20 client.4113.objecter shutdown clearing up homeless session... 2023-10-30T17:31:41.078 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 10 client.4113.objecter successfully canceled tick 2023-10-30T17:31:41.079 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 10 monclient: shutdown 2023-10-30T17:31:41.079 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:41.079 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 1 -- 172.21.15.177:0/2681568334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132413f3d0 msgr2=0x7f13241cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.079 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 1 --2- 172.21.15.177:0/2681568334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132413f3d0 0x7f13241cc840 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f130c033000 tx=0x7f130c0046c0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.080 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 1 -- 172.21.15.177:0/2681568334 shutdown_connections 2023-10-30T17:31:41.080 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 1 --2- 172.21.15.177:0/2681568334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1324141ae0 0x7f13240ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.080 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 1 --2- 172.21.15.177:0/2681568334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132413f3d0 0x7f13241cc840 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.080 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 1 --2- 172.21.15.177:0/2681568334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13240b4500 0x7f132413b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.080 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 1 -- 172.21.15.177:0/2681568334 >> 172.21.15.177:0/2681568334 conn(0x7f1324071b60 msgr2=0x7f1324071f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:41.080 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 1 -- 172.21.15.177:0/2681568334 shutdown_connections 2023-10-30T17:31:41.081 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 1 -- 172.21.15.177:0/2681568334 wait complete. 2023-10-30T17:31:41.081 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.075+0000 7f132987c640 1 librados: shutdown 2023-10-30T17:31:41.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be 1 2023-10-30T17:31:41.174 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:41.173+0000 7f2fb4c2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T17:31:41.186 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.184+0000 7f67037c9640 10 monclient: get_monmap_and_config 2023-10-30T17:31:41.186 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.184+0000 7f67037c9640 10 monclient: build_initial_monmap 2023-10-30T17:31:41.186 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.184+0000 7f67037c9640 10 monclient: monmap: 2023-10-30T17:31:41.187 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:41.187 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:41.187 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:31:41.185616+0000 2023-10-30T17:31:41.187 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:31:41.185616+0000 2023-10-30T17:31:41.188 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:41.188 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.188 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:41.188 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:41.189 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:41.189 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:41.189 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:41.189 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.189 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 1 Processor -- start 2023-10-30T17:31:41.193 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 1 -- start start 2023-10-30T17:31:41.193 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 10 monclient: init 2023-10-30T17:31:41.193 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:41.194 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-10-30T17:31:41.194 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66fc06a2b0 addr v1:172.21.15.170:6789/0 2023-10-30T17:31:41.194 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f66fc0af060 0x7f66fc071060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:41.194 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 10 monclient(hunting): picked mon.noname-a con 0x7f66fc0af060 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:41.195 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc0715a0 0x7f66fc14c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:41.195 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 10 monclient(hunting): picked mon.noname-b con 0x7f66fc0715a0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:41.195 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.185+0000 7f67037c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66fc0aebd0 con 0x7f66fc06a2b0 2023-10-30T17:31:41.195 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.186+0000 7f67037c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:41.196 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.186+0000 7f67037c9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66fc06a690 con 0x7f66fc0af060 2023-10-30T17:31:41.196 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.186+0000 7f67037c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:41.196 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.186+0000 7f67037c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc0b04a0 con 0x7f66fc0715a0 2023-10-30T17:31:41.196 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.186+0000 7f67037c9640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:41.197 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.186+0000 7f67037c9640 10 monclient(hunting): authenticate will time out at 1274.547050s 2023-10-30T17:31:41.197 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.186+0000 7f6700d64640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f66fc0af060 0x7f66fc071060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:41.197 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.186+0000 7f6701d66640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc0715a0 0x7f66fc14c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:41.197 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.187+0000 7f6701565640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f66fc06a2b0 0x7f66fc141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.177:47176/0 (socket says 172.21.15.177:47176) 2023-10-30T17:31:41.198 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.187+0000 7f6701565640 1 -- 172.21.15.177:0/1163774754 learned_addr learned my addr 172.21.15.177:0/1163774754 (peer_addr_for_me v1:172.21.15.177:0/0) 2023-10-30T17:31:41.198 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.187+0000 7f6701d66640 10 monclient(hunting): get_auth_request con 0x7f66fc0715a0 auth_method 0 2023-10-30T17:31:41.198 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.187+0000 7f6701d66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:41.198 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.187+0000 7f6701d66640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.199 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.187+0000 7f6701d66640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.199 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.187+0000 7f6700d64640 10 monclient(hunting): get_auth_request con 0x7f66fc0af060 auth_method 0 2023-10-30T17:31:41.199 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.187+0000 7f6700d64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:41.199 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.187+0000 7f6700d64640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.200 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.187+0000 7f6700d64640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.200 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.188+0000 7f6701d66640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:41.200 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.188+0000 7f6701d66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:41.200 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.188+0000 7f6701d66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:41.201 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.188+0000 7f6700d64640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:41.201 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.188+0000 7f6700d64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:41.201 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.188+0000 7f6700d64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:41.201 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.190+0000 7f6701d66640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2023-10-30T17:31:41.201 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.190+0000 7f6701d66640 1 -- 172.21.15.177:0/1163774754 >> v2:172.21.15.170:3300/0 conn(0x7f66fc0af060 msgr2=0x7f66fc071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:41.202 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.190+0000 7f6701d66640 1 --2- 172.21.15.177:0/1163774754 >> v2:172.21.15.170:3300/0 conn(0x7f66fc0af060 0x7f66fc071060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.202 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.190+0000 7f6701d66640 1 -- 172.21.15.177:0/1163774754 >> v1:172.21.15.170:6789/0 conn(0x7f66fc06a2b0 legacy=0x7f66fc141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.202 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.190+0000 7f6701d66640 10 monclient: _finish_hunting 0 2023-10-30T17:31:41.202 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.190+0000 7f6701d66640 1 monclient: found mon.noname-b 2023-10-30T17:31:41.202 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.190+0000 7f6701d66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:41.203 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.190+0000 7f6701d66640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:31:41.203 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.190+0000 7f6701d66640 1 -- 172.21.15.177:0/1163774754 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc1421b0 con 0x7f66fc0715a0 2023-10-30T17:31:41.203 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.194+0000 7f6701d66640 1 --2- 172.21.15.177:0/1163774754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc0715a0 0x7f66fc14c3b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009d90 tx=0x7f66f402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b30d181d754804f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:41.235 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:41.233+0000 7f2fb4c2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T17:31:41.236 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 1 -- 172.21.15.177:0/1163774754 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f402cc50 con 0x7f66fc0715a0 2023-10-30T17:31:41.236 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:41.237 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:31:41.237 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 10 monclient: dump: 2023-10-30T17:31:41.237 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:41.237 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:41.237 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.238 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.238 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:41.238 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.238 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:41.239 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:41.239 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:41.239 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.239 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 10 monclient: _finish_auth 0 2023-10-30T17:31:41.240 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 10 monclient: _check_auth_tickets 2023-10-30T17:31:41.240 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:41.240 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 1 -- 172.21.15.177:0/1163774754 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f402cdb0 con 0x7f66fc0715a0 2023-10-30T17:31:41.240 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:41.240 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 1 -- 172.21.15.177:0/1163774754 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4034660 con 0x7f66fc0715a0 2023-10-30T17:31:41.241 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:41.241 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:31:41.241 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f66f3fff640 10 monclient: dump: 2023-10-30T17:31:41.241 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:41.242 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:41.242 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.242 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.242 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:41.242 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.243 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:41.243 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:41.243 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:41.243 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.243 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f67037c9640 5 monclient: authenticate success, global_id 4112 2023-10-30T17:31:41.244 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f67037c9640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:41.244 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f67037c9640 4 set_mon_vals no callback set 2023-10-30T17:31:41.245 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f67037c9640 10 monclient: shutdown 2023-10-30T17:31:41.245 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f67037c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:41.245 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f67037c9640 1 -- 172.21.15.177:0/1163774754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc0715a0 msgr2=0x7f66fc14c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.245 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.235+0000 7f67037c9640 1 --2- 172.21.15.177:0/1163774754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc0715a0 0x7f66fc14c3b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009d90 tx=0x7f66f402bea0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.246 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.236+0000 7f67037c9640 1 -- 172.21.15.177:0/1163774754 shutdown_connections 2023-10-30T17:31:41.246 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.236+0000 7f67037c9640 1 --2- 172.21.15.177:0/1163774754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc0715a0 0x7f66fc14c3b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.246 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.236+0000 7f67037c9640 1 --2- 172.21.15.177:0/1163774754 >> v2:172.21.15.170:3300/0 conn(0x7f66fc0af060 0x7f66fc071060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.246 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.236+0000 7f67037c9640 1 -- 172.21.15.177:0/1163774754 >> 172.21.15.177:0/1163774754 conn(0x7f66fc0b4020 msgr2=0x7f66fc0b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:41.247 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.236+0000 7f67037c9640 1 -- 172.21.15.177:0/1163774754 shutdown_connections 2023-10-30T17:31:41.247 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.236+0000 7f67037c9640 1 -- 172.21.15.177:0/1163774754 wait complete. 2023-10-30T17:31:41.247 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:41.236+0000 7f2b9a22b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T17:31:41.250 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.236+0000 7f67037c9640 10 monclient: build_initial_monmap 2023-10-30T17:31:41.251 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.236+0000 7f67037c9640 10 monclient: monmap: 2023-10-30T17:31:41.251 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:41.251 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:41.251 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2023-10-30T17:31:41.251 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2023-10-30T17:31:41.252 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:41.252 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.252 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:41.252 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:41.252 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:41.253 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.253 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 librados: starting msgr at 2023-10-30T17:31:41.253 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 librados: starting objecter 2023-10-30T17:31:41.253 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 Processor -- start 2023-10-30T17:31:41.253 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 -- start start 2023-10-30T17:31:41.254 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 librados: setting wanted keys 2023-10-30T17:31:41.254 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 librados: calling monclient init 2023-10-30T17:31:41.254 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient: init 2023-10-30T17:31:41.254 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:41.254 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:31:41.255 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc06a2b0 0x7f66fc13aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:41.255 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient(hunting): picked mon.noname-c con 0x7f66fc06a2b0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:41.255 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66fc0715a0 0x7f66fc1cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:41.255 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient(hunting): picked mon.noname-a con 0x7f66fc0715a0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:41.256 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc0af060 0x7f66fc13e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:41.256 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient(hunting): picked mon.noname-b con 0x7f66fc0af060 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:41.256 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:41.256 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc141700 con 0x7f66fc0715a0 2023-10-30T17:31:41.256 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:41.256 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc1419a0 con 0x7f66fc0af060 2023-10-30T17:31:41.257 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:41.257 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc13e120 con 0x7f66fc06a2b0 2023-10-30T17:31:41.257 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:41.257 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.237+0000 7f67037c9640 10 monclient(hunting): authenticate will time out at 1274.598837s 2023-10-30T17:31:41.257 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.238+0000 7f6701d66640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc0af060 0x7f66fc13e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:41.258 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.238+0000 7f6700d64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66fc0715a0 0x7f66fc1cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:41.258 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.238+0000 7f6700d64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66fc0715a0 0x7f66fc1cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.177:44306/0 (socket says 172.21.15.177:44306) 2023-10-30T17:31:41.258 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.238+0000 7f6700d64640 1 -- 172.21.15.177:0/3060128995 learned_addr learned my addr 172.21.15.177:0/3060128995 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:41.258 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.238+0000 7f6700d64640 10 monclient(hunting): get_auth_request con 0x7f66fc0715a0 auth_method 0 2023-10-30T17:31:41.259 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.238+0000 7f6700d64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:41.259 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.238+0000 7f6700d64640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.259 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.238+0000 7f6700d64640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.259 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.239+0000 7f6701d66640 10 monclient(hunting): get_auth_request con 0x7f66fc0af060 auth_method 0 2023-10-30T17:31:41.259 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.239+0000 7f6701d66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:41.260 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.239+0000 7f6701d66640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.260 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.239+0000 7f6701d66640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.260 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.239+0000 7f6700d64640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:41.260 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.239+0000 7f6700d64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:41.260 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.239+0000 7f6700d64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:41.261 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.240+0000 7f6701d66640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:41.261 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.240+0000 7f6701d66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:41.261 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.240+0000 7f6701d66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:41.261 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701565640 1 --2- 172.21.15.177:0/3060128995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc06a2b0 0x7f66fc13aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:41.262 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701565640 10 monclient(hunting): get_auth_request con 0x7f66fc06a2b0 auth_method 0 2023-10-30T17:31:41.262 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701565640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:41.262 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701565640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.262 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701565640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.263 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701d66640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2023-10-30T17:31:41.263 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701d66640 1 -- 172.21.15.177:0/3060128995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc06a2b0 msgr2=0x7f66fc13aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.263 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701d66640 1 --2- 172.21.15.177:0/3060128995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc06a2b0 0x7f66fc13aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.263 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701d66640 1 -- 172.21.15.177:0/3060128995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66fc0715a0 msgr2=0x7f66fc1cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.264 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701d66640 1 --2- 172.21.15.177:0/3060128995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66fc0715a0 0x7f66fc1cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.264 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701d66640 10 monclient: _finish_hunting 0 2023-10-30T17:31:41.264 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701d66640 1 monclient: found mon.noname-b 2023-10-30T17:31:41.265 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701d66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:41.265 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701d66640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:31:41.265 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.243+0000 7f6701d66640 1 -- 172.21.15.177:0/3060128995 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f4009a70 con 0x7f66fc0af060 2023-10-30T17:31:41.265 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.246+0000 7f6701d66640 1 --2- 172.21.15.177:0/3060128995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc0af060 0x7f66fc13e7b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f66f4033000 tx=0x7f66f4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:41.266 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.246+0000 7f66f1ffb640 10 client.?.objecter ms_handle_connect 0x7f66fc0af060 2023-10-30T17:31:41.266 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.246+0000 7f66f1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:41.267 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 1 -- 172.21.15.177:0/3060128995 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4040650 con 0x7f66fc0af060 2023-10-30T17:31:41.268 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:41.268 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:31:41.268 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 10 monclient: dump: 2023-10-30T17:31:41.268 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:41.269 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:41.269 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.269 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.269 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:41.269 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.270 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:41.270 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:41.270 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:41.270 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.271 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 10 monclient: _finish_auth 0 2023-10-30T17:31:41.271 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:31:41.271 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:41.271 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 1 -- 172.21.15.177:0/3060128995 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f40407b0 con 0x7f66fc0af060 2023-10-30T17:31:41.271 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:41.272 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 1 -- 172.21.15.177:0/3060128995 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4040b60 con 0x7f66fc0af060 2023-10-30T17:31:41.272 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.249+0000 7f66f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:41.272 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f66f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:31:41.272 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f66f1ffb640 10 monclient: dump: 2023-10-30T17:31:41.273 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:41.273 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:41.273 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.273 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.273 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:41.274 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.274 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:41.274 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:41.274 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:41.275 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.275 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f67037c9640 5 monclient: authenticate success, global_id 4120 2023-10-30T17:31:41.275 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f67037c9640 10 monclient: _renew_subs 2023-10-30T17:31:41.275 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f66f2ffd640 4 set_mon_vals no callback set 2023-10-30T17:31:41.275 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f67037c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:41.276 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f67037c9640 1 -- 172.21.15.177:0/3060128995 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66fc140ae0 con 0x7f66fc0af060 2023-10-30T17:31:41.276 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f67037c9640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:41.276 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f67037c9640 10 monclient: _renew_subs 2023-10-30T17:31:41.276 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f67037c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:41.277 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f67037c9640 1 -- 172.21.15.177:0/3060128995 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66fc13eed0 con 0x7f66fc0af060 2023-10-30T17:31:41.277 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.250+0000 7f67037c9640 1 librados: init done 2023-10-30T17:31:41.277 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.251+0000 7f67037c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:41.277 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.251+0000 7f67037c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:41.277 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.251+0000 7f67037c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:41.278 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.251+0000 7f67037c9640 1 -- 172.21.15.177:0/3060128995 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66fc13e540 con 0x7f66fc0af060 2023-10-30T17:31:41.278 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.252+0000 7f66f1ffb640 1 -- 172.21.15.177:0/3060128995 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f66f4052430 con 0x7f66fc0af060 2023-10-30T17:31:41.278 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.254+0000 7f66f1ffb640 1 -- 172.21.15.177:0/3060128995 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f66f40497b0 con 0x7f66fc0af060 2023-10-30T17:31:41.278 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.254+0000 7f66f1ffb640 10 client.4120.objecter ms_dispatch 0x7f66fc000b90 osd_map(3..3 src has 1..3) v4 2023-10-30T17:31:41.279 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.254+0000 7f66f1ffb640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2023-10-30T17:31:41.279 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.254+0000 7f66f1ffb640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2023-10-30T17:31:41.279 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.254+0000 7f66f1ffb640 20 client.4120.objecter dump_active .. 0 homeless 2023-10-30T17:31:41.285 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:41.284+0000 7f2fb4c2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T17:31:41.293 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:41.292+0000 7f2b9a22b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T17:31:41.339 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:41.337+0000 7f2fb4c2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T17:31:41.340 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:41.338+0000 7f2b9a22b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T17:31:41.354 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.353+0000 7f66f1ffb640 1 -- 172.21.15.177:0/3060128995 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f66f4044020 con 0x7f66fc0af060 2023-10-30T17:31:41.354 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.353+0000 7f66f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:41.355 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.353+0000 7f66f1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:41.381 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:41.379+0000 7f2fb4c2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T17:31:41.395 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:41.393+0000 7f2b9a22b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T17:31:41.426 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.424+0000 7f67037c9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be", "id": 1}] 2023-10-30T17:31:41.426 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.424+0000 7f67037c9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be", "id": 1}] 2023-10-30T17:31:41.426 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.424+0000 7f67037c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:41.427 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.424+0000 7f67037c9640 1 -- 172.21.15.177:0/3060128995 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be", "id": 1} v 0) v1 -- 0x7f66fc067cd0 con 0x7f66fc0af060 2023-10-30T17:31:41.438 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:41.437+0000 7f2b9a22b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T17:31:41.484 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:41.483+0000 7f2fb4c2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-30T17:31:41.512 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f66f1ffb640 1 -- 172.21.15.177:0/3060128995 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f66f4039070 con 0x7f66fc0af060 2023-10-30T17:31:41.513 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f66f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be", "id": 1}] 2023-10-30T17:31:41.513 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f66f1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:41.514 INFO:teuthology.orchestra.run.smithi177.stdout:1 2023-10-30T17:31:41.515 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f67037c9640 10 librados: watch_flush enter 2023-10-30T17:31:41.516 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f67037c9640 10 librados: watch_flush exit 2023-10-30T17:31:41.516 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f67037c9640 20 client.4120.objecter shutdown clearing up homeless session... 2023-10-30T17:31:41.516 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f67037c9640 10 client.4120.objecter successfully canceled tick 2023-10-30T17:31:41.516 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f67037c9640 10 monclient: shutdown 2023-10-30T17:31:41.517 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f67037c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:41.517 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f67037c9640 1 -- 172.21.15.177:0/3060128995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc0af060 msgr2=0x7f66fc13e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.517 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.511+0000 7f67037c9640 1 --2- 172.21.15.177:0/3060128995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc0af060 0x7f66fc13e7b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f66f4033000 tx=0x7f66f4004a40 comp rx=0 tx=0).stop 2023-10-30T17:31:41.517 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.512+0000 7f67037c9640 1 -- 172.21.15.177:0/3060128995 shutdown_connections 2023-10-30T17:31:41.517 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.512+0000 7f67037c9640 1 --2- 172.21.15.177:0/3060128995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc0af060 0x7f66fc13e7b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.518 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.512+0000 7f67037c9640 1 --2- 172.21.15.177:0/3060128995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66fc0715a0 0x7f66fc1cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.518 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.512+0000 7f67037c9640 1 --2- 172.21.15.177:0/3060128995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66fc06a2b0 0x7f66fc13aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.518 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.512+0000 7f67037c9640 1 -- 172.21.15.177:0/3060128995 >> 172.21.15.177:0/3060128995 conn(0x7f66fc0b4020 msgr2=0x7f66fc0b0900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:41.518 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.512+0000 7f67037c9640 1 -- 172.21.15.177:0/3060128995 shutdown_connections 2023-10-30T17:31:41.519 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.512+0000 7f67037c9640 1 -- 172.21.15.177:0/3060128995 wait complete. 2023-10-30T17:31:41.519 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.512+0000 7f67037c9640 1 librados: shutdown 2023-10-30T17:31:41.523 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 8f70bfd6-88a2-413b-9663-f9bfed6ebe74 2 2023-10-30T17:31:41.545 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:41.543+0000 7f2b9a22b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-30T17:31:41.556 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:41.555+0000 7f2fb4c2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T17:31:41.601 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:41.600+0000 7f2fb4c2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T17:31:41.618 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:41.616+0000 7f2b9a22b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T17:31:41.622 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.621+0000 7f99022b4640 10 monclient: get_monmap_and_config 2023-10-30T17:31:41.623 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.621+0000 7f99022b4640 10 monclient: build_initial_monmap 2023-10-30T17:31:41.623 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.621+0000 7f99022b4640 10 monclient: monmap: 2023-10-30T17:31:41.623 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:41.623 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:41.624 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:31:41.622594+0000 2023-10-30T17:31:41.624 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:31:41.622594+0000 2023-10-30T17:31:41.624 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:41.624 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.624 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:41.625 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:41.625 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:41.625 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:41.625 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:41.625 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.628 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 1 Processor -- start 2023-10-30T17:31:41.628 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 1 -- start start 2023-10-30T17:31:41.628 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 10 monclient: init 2023-10-30T17:31:41.629 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:41.629 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-10-30T17:31:41.629 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f98fc071900 0x7f98fc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:41.629 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 10 monclient(hunting): picked mon.noname-a con 0x7f98fc071900 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:41.630 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98fc06e4d0 0x7f98fc0ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:41.630 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 10 monclient(hunting): picked mon.noname-b con 0x7f98fc06e4d0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:41.630 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98fc06f3f0 addr v1:172.21.15.177:6789/0 2023-10-30T17:31:41.631 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc071470 con 0x7f98fc06f3f0 2023-10-30T17:31:41.631 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:41.631 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98fc13c420 con 0x7f98fc071900 2023-10-30T17:31:41.631 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:41.631 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc072220 con 0x7f98fc06e4d0 2023-10-30T17:31:41.632 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:41.632 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.622+0000 7f99022b4640 10 monclient(hunting): authenticate will time out at 1274.983968s 2023-10-30T17:31:41.632 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.623+0000 7f98fb7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98fc06e4d0 0x7f98fc0ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:41.632 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.623+0000 7f98fbfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f98fc071900 0x7f98fc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:41.632 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.623+0000 7f9900851640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f98fc06f3f0 0x7f98fc13e760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:38238/0 (socket says 172.21.15.177:38238) 2023-10-30T17:31:41.633 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.623+0000 7f9900851640 1 -- 172.21.15.177:0/362267769 learned_addr learned my addr 172.21.15.177:0/362267769 (peer_addr_for_me v1:172.21.15.177:0/0) 2023-10-30T17:31:41.633 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fbfff640 10 monclient(hunting): get_auth_request con 0x7f98fc071900 auth_method 0 2023-10-30T17:31:41.633 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:41.633 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.634 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.634 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fb7fe640 10 monclient(hunting): get_auth_request con 0x7f98fc06e4d0 auth_method 0 2023-10-30T17:31:41.634 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:41.634 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.634 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.635 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:41.635 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:41.635 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:41.635 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:41.635 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:41.636 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.624+0000 7f98fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:41.636 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98faffd640 1 -- 172.21.15.177:0/362267769 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153419115 0 0) 0x7f98fc071470 con 0x7f98fc06f3f0 2023-10-30T17:31:41.636 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98faffd640 10 monclient(hunting): _negotiate 2023-10-30T17:31:41.636 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98faffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.636 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98faffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.637 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98faffd640 10 monclient(hunting): my global_id is 4123 2023-10-30T17:31:41.637 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98faffd640 1 -- 172.21.15.177:0/362267769 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e4003520 con 0x7f98fc06f3f0 2023-10-30T17:31:41.637 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98fbfff640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2023-10-30T17:31:41.637 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98fbfff640 1 -- 172.21.15.177:0/362267769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98fc06e4d0 msgr2=0x7f98fc0ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:41.637 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98fbfff640 1 --2- 172.21.15.177:0/362267769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98fc06e4d0 0x7f98fc0ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.638 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98fbfff640 1 -- 172.21.15.177:0/362267769 >> v1:172.21.15.177:6789/0 conn(0x7f98fc06f3f0 legacy=0x7f98fc13e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.638 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98fbfff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:41.638 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98fbfff640 1 monclient: found mon.noname-a 2023-10-30T17:31:41.638 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:41.639 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:41.639 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.626+0000 7f98fbfff640 1 -- 172.21.15.177:0/362267769 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc06ebf0 con 0x7f98fc071900 2023-10-30T17:31:41.645 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.628+0000 7f98fbfff640 1 --2- 172.21.15.177:0/362267769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc071900 0x7f98fc071ce0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f98ec0097d0 tx=0x7f98ec02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848a40810efafbc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:41.646 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 1 -- 172.21.15.177:0/362267769 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec031770 con 0x7f98fc071900 2023-10-30T17:31:41.646 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:41.646 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:41.647 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 10 monclient: dump: 2023-10-30T17:31:41.647 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:41.647 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:41.648 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.648 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.648 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:41.648 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.649 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:41.649 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:41.649 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:41.649 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.649 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 10 monclient: _finish_auth 0 2023-10-30T17:31:41.649 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 10 monclient: _check_auth_tickets 2023-10-30T17:31:41.650 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:41.650 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 1 -- 172.21.15.177:0/362267769 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec0318d0 con 0x7f98fc071900 2023-10-30T17:31:41.650 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f99022b4640 5 monclient: authenticate success, global_id 4125 2023-10-30T17:31:41.650 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f99022b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:31:41.651 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:41.651 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 1 -- 172.21.15.177:0/362267769 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec031c60 con 0x7f98fc071900 2023-10-30T17:31:41.651 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f99022b4640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:41.651 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f99022b4640 4 set_mon_vals no callback set 2023-10-30T17:31:41.652 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:41.652 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:41.652 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.631+0000 7f98faffd640 10 monclient: dump: 2023-10-30T17:31:41.652 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:41.652 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:41.652 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.653 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.653 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:41.653 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.653 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:41.653 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:41.654 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:41.654 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.654 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.632+0000 7f99022b4640 10 monclient: shutdown 2023-10-30T17:31:41.654 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.632+0000 7f99022b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:41.654 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.632+0000 7f99022b4640 1 -- 172.21.15.177:0/362267769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc071900 msgr2=0x7f98fc071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.655 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.632+0000 7f99022b4640 1 --2- 172.21.15.177:0/362267769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc071900 0x7f98fc071ce0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f98ec0097d0 tx=0x7f98ec02b810 comp rx=0 tx=0).stop 2023-10-30T17:31:41.655 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.632+0000 7f99022b4640 1 -- 172.21.15.177:0/362267769 shutdown_connections 2023-10-30T17:31:41.655 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.632+0000 7f99022b4640 1 --2- 172.21.15.177:0/362267769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98fc06e4d0 0x7f98fc0ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.655 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.632+0000 7f99022b4640 1 --2- 172.21.15.177:0/362267769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc071900 0x7f98fc071ce0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.655 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.632+0000 7f99022b4640 1 -- 172.21.15.177:0/362267769 >> 172.21.15.177:0/362267769 conn(0x7f98fc0b4020 msgr2=0x7f98fc0b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:41.656 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.632+0000 7f99022b4640 1 -- 172.21.15.177:0/362267769 shutdown_connections 2023-10-30T17:31:41.656 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.632+0000 7f99022b4640 1 -- 172.21.15.177:0/362267769 wait complete. 2023-10-30T17:31:41.657 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.633+0000 7f99022b4640 10 monclient: build_initial_monmap 2023-10-30T17:31:41.657 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.633+0000 7f99022b4640 10 monclient: monmap: 2023-10-30T17:31:41.657 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:41.657 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:41.657 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2023-10-30T17:31:41.658 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2023-10-30T17:31:41.658 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:41.658 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.658 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:41.659 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:41.659 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:41.659 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.659 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.633+0000 7f99022b4640 1 librados: starting msgr at 2023-10-30T17:31:41.659 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.633+0000 7f99022b4640 1 librados: starting objecter 2023-10-30T17:31:41.660 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.634+0000 7f99022b4640 1 Processor -- start 2023-10-30T17:31:41.660 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.634+0000 7f99022b4640 1 -- start start 2023-10-30T17:31:41.660 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.634+0000 7f99022b4640 1 librados: setting wanted keys 2023-10-30T17:31:41.660 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.634+0000 7f99022b4640 1 librados: calling monclient init 2023-10-30T17:31:41.660 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.634+0000 7f99022b4640 10 monclient: init 2023-10-30T17:31:41.660 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:41.661 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:31:41.661 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc06e4d0 0x7f98fc1420f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:41.662 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 10 monclient(hunting): picked mon.noname-a con 0x7f98fc06e4d0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:41.662 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98fc06f3f0 0x7f98fc13d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:41.663 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 10 monclient(hunting): picked mon.noname-c con 0x7f98fc06f3f0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:41.663 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98fc071900 0x7f98fc1d2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:41.663 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 10 monclient(hunting): picked mon.noname-b con 0x7f98fc071900 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:41.663 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:41.664 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc142900 con 0x7f98fc06e4d0 2023-10-30T17:31:41.664 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:41.665 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc141c60 con 0x7f98fc071900 2023-10-30T17:31:41.665 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:41.665 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc1418f0 con 0x7f98fc06f3f0 2023-10-30T17:31:41.666 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:41.666 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f99022b4640 10 monclient(hunting): authenticate will time out at 1274.996219s 2023-10-30T17:31:41.666 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f98fb7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98fc06f3f0 0x7f98fc13d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:41.667 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f98fbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc06e4d0 0x7f98fc1420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:41.667 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.635+0000 7f9900851640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98fc071900 0x7f98fc1d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:41.667 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc06e4d0 0x7f98fc1420f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.177:44318/0 (socket says 172.21.15.177:44318) 2023-10-30T17:31:41.668 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fbfff640 1 -- 172.21.15.177:0/3731567152 learned_addr learned my addr 172.21.15.177:0/3731567152 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:41.668 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fbfff640 10 monclient(hunting): get_auth_request con 0x7f98fc06e4d0 auth_method 0 2023-10-30T17:31:41.668 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:41.669 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.669 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.670 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f9900851640 10 monclient(hunting): get_auth_request con 0x7f98fc071900 auth_method 0 2023-10-30T17:31:41.670 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f9900851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:41.670 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f9900851640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.671 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f9900851640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.671 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fb7fe640 10 monclient(hunting): get_auth_request con 0x7f98fc06f3f0 auth_method 0 2023-10-30T17:31:41.671 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:41.672 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:41.672 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:41.672 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:41.672 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:41.672 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:41.672 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f9900851640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:41.673 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f9900851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:41.673 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f9900851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:41.673 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:41.673 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:41.673 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.636+0000 7f98fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:41.674 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.639+0000 7f98fbfff640 10 monclient(hunting): handle_auth_done global_id 4128 payload 824 2023-10-30T17:31:41.674 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.639+0000 7f98fbfff640 1 -- 172.21.15.177:0/3731567152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98fc06f3f0 msgr2=0x7f98fc13d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.674 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.639+0000 7f98fbfff640 1 --2- 172.21.15.177:0/3731567152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98fc06f3f0 0x7f98fc13d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.674 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.639+0000 7f98fbfff640 1 -- 172.21.15.177:0/3731567152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98fc071900 msgr2=0x7f98fc1d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.674 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.639+0000 7f98fbfff640 1 --2- 172.21.15.177:0/3731567152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98fc071900 0x7f98fc1d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.675 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.639+0000 7f98fbfff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:41.675 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.639+0000 7f98fbfff640 1 monclient: found mon.noname-a 2023-10-30T17:31:41.675 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.639+0000 7f98fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:41.675 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.639+0000 7f98fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:41.675 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.639+0000 7f98fbfff640 1 -- 172.21.15.177:0/3731567152 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec009400 con 0x7f98fc06e4d0 2023-10-30T17:31:41.676 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.641+0000 7f98fbfff640 1 --2- 172.21.15.177:0/3731567152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc06e4d0 0x7f98fc1420f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f98ec034040 tx=0x7f98ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:41.676 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.641+0000 7f98f8ff9640 10 client.?.objecter ms_handle_connect 0x7f98fc06e4d0 2023-10-30T17:31:41.676 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.641+0000 7f98f8ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:41.677 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:41.655+0000 7f2fb4c2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T17:31:41.678 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 1 -- 172.21.15.177:0/3731567152 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec032610 con 0x7f98fc06e4d0 2023-10-30T17:31:41.679 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:41.679 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:41.679 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 10 monclient: dump: 2023-10-30T17:31:41.679 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:41.679 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:41.680 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.680 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.680 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:41.680 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.680 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:41.680 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:41.681 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:41.681 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.681 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 10 monclient: _finish_auth 0 2023-10-30T17:31:41.681 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 10 monclient: _check_auth_tickets 2023-10-30T17:31:41.681 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:41.682 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 1 -- 172.21.15.177:0/3731567152 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec032770 con 0x7f98fc06e4d0 2023-10-30T17:31:41.682 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:41.682 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 1 -- 172.21.15.177:0/3731567152 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec032b20 con 0x7f98fc06e4d0 2023-10-30T17:31:41.682 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f99022b4640 5 monclient: authenticate success, global_id 4128 2023-10-30T17:31:41.682 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f99022b4640 10 monclient: _renew_subs 2023-10-30T17:31:41.683 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f99022b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:41.683 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f99022b4640 1 -- 172.21.15.177:0/3731567152 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc1d2c60 con 0x7f98fc06e4d0 2023-10-30T17:31:41.683 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98fa7fc640 4 set_mon_vals no callback set 2023-10-30T17:31:41.683 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:41.684 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:41.684 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f98f8ff9640 10 monclient: dump: 2023-10-30T17:31:41.684 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:41.684 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:41.684 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.685 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:41.685 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:41.685 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:41.685 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:41.685 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:41.686 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:41.686 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:41.686 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f99022b4640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:41.689 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f99022b4640 10 monclient: _renew_subs 2023-10-30T17:31:41.689 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f99022b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f99022b4640 1 -- 172.21.15.177:0/3731567152 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc141610 con 0x7f98fc06e4d0 2023-10-30T17:31:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.645+0000 7f99022b4640 1 librados: init done 2023-10-30T17:31:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.646+0000 7f99022b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.646+0000 7f99022b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.646+0000 7f99022b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:41.691 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.646+0000 7f99022b4640 1 -- 172.21.15.177:0/3731567152 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc141ab0 con 0x7f98fc06e4d0 2023-10-30T17:31:41.691 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.647+0000 7f98f8ff9640 1 -- 172.21.15.177:0/3731567152 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f98ec005990 con 0x7f98fc06e4d0 2023-10-30T17:31:41.691 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.647+0000 7f98f8ff9640 1 -- 172.21.15.177:0/3731567152 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f98ec04e440 con 0x7f98fc06e4d0 2023-10-30T17:31:41.691 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.647+0000 7f98f8ff9640 10 client.4128.objecter ms_dispatch 0x7f98fc000b90 osd_map(4..4 src has 1..4) v4 2023-10-30T17:31:41.691 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.647+0000 7f98f8ff9640 3 client.4128.objecter handle_osd_map got epochs [4,4] > 0 2023-10-30T17:31:41.692 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.647+0000 7f98f8ff9640 3 client.4128.objecter handle_osd_map decoding full epoch 4 2023-10-30T17:31:41.692 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.647+0000 7f98f8ff9640 20 client.4128.objecter dump_active .. 0 homeless 2023-10-30T17:31:41.692 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:41.666+0000 7f2b9a22b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T17:31:41.705 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:41.703+0000 7f2fb4c2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T17:31:41.719 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:41.718+0000 7f2b9a22b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T17:31:41.759 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.758+0000 7f98f8ff9640 1 -- 172.21.15.177:0/3731567152 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f98ec0065c0 con 0x7f98fc06e4d0 2023-10-30T17:31:41.760 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.758+0000 7f98f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:41.760 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.758+0000 7f98f8ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:41.765 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:41.764+0000 7f2b9a22b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T17:31:41.853 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.851+0000 7f99022b4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8f70bfd6-88a2-413b-9663-f9bfed6ebe74", "id": 2}] 2023-10-30T17:31:41.853 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.851+0000 7f99022b4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8f70bfd6-88a2-413b-9663-f9bfed6ebe74", "id": 2}] 2023-10-30T17:31:41.853 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.851+0000 7f99022b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:41.854 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.851+0000 7f99022b4640 1 -- 172.21.15.177:0/3731567152 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8f70bfd6-88a2-413b-9663-f9bfed6ebe74", "id": 2} v 0) v1 -- 0x7f98fc067cd0 con 0x7f98fc06e4d0 2023-10-30T17:31:41.906 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.904+0000 7f98f8ff9640 1 -- 172.21.15.177:0/3731567152 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f70bfd6-88a2-413b-9663-f9bfed6ebe74", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f98ec005c40 con 0x7f98fc06e4d0 2023-10-30T17:31:41.906 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.904+0000 7f98f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8f70bfd6-88a2-413b-9663-f9bfed6ebe74", "id": 2}] 2023-10-30T17:31:41.906 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.904+0000 7f98f8ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:41.907 INFO:teuthology.orchestra.run.smithi177.stdout:2 2023-10-30T17:31:41.908 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 10 librados: watch_flush enter 2023-10-30T17:31:41.909 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 10 librados: watch_flush exit 2023-10-30T17:31:41.909 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 20 client.4128.objecter shutdown clearing up homeless session... 2023-10-30T17:31:41.909 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 10 client.4128.objecter successfully canceled tick 2023-10-30T17:31:41.909 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 10 monclient: shutdown 2023-10-30T17:31:41.909 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:41.910 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 1 -- 172.21.15.177:0/3731567152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc06e4d0 msgr2=0x7f98fc1420f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:41.910 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 1 --2- 172.21.15.177:0/3731567152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc06e4d0 0x7f98fc1420f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f98ec034040 tx=0x7f98ec0046c0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.910 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 1 -- 172.21.15.177:0/3731567152 shutdown_connections 2023-10-30T17:31:41.910 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 1 --2- 172.21.15.177:0/3731567152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98fc071900 0x7f98fc1d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.911 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 1 --2- 172.21.15.177:0/3731567152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98fc06f3f0 0x7f98fc13d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.911 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 1 --2- 172.21.15.177:0/3731567152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98fc06e4d0 0x7f98fc1420f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:41.911 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 1 -- 172.21.15.177:0/3731567152 >> 172.21.15.177:0/3731567152 conn(0x7f98fc0b4020 msgr2=0x7f98fc06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:41.911 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 1 -- 172.21.15.177:0/3731567152 shutdown_connections 2023-10-30T17:31:41.911 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.905+0000 7f99022b4640 1 -- 172.21.15.177:0/3731567152 wait complete. 2023-10-30T17:31:41.912 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:41.906+0000 7f99022b4640 1 librados: shutdown 2023-10-30T17:31:41.918 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 46c2dd9a-c073-42ab-a07b-918f83b8f7f6 3 2023-10-30T17:31:42.058 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.057+0000 7f9705141640 10 monclient: get_monmap_and_config 2023-10-30T17:31:42.059 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.057+0000 7f9705141640 10 monclient: build_initial_monmap 2023-10-30T17:31:42.059 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.057+0000 7f9705141640 10 monclient: monmap: 2023-10-30T17:31:42.059 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:42.059 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.060 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:31:42.058158+0000 2023-10-30T17:31:42.060 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:31:42.058158+0000 2023-10-30T17:31:42.060 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:42.060 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.060 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:42.061 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:42.061 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:42.061 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:42.061 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:42.061 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.062 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.057+0000 7f9705141640 1 Processor -- start 2023-10-30T17:31:42.062 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.057+0000 7f9705141640 1 -- start start 2023-10-30T17:31:42.062 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.057+0000 7f9705141640 10 monclient: init 2023-10-30T17:31:42.065 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:42.065 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T17:31:42.066 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f97000b4500 0x7f97000a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.066 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 10 monclient(hunting): picked mon.noname-c con 0x7f97000b4500 addr v2:172.21.15.177:3300/0 2023-10-30T17:31:42.066 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f970013ecc0 0x7f970013f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.066 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 10 monclient(hunting): picked mon.noname-a con 0x7f970013ecc0 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:42.066 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f9700141f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.067 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 10 monclient(hunting): picked mon.noname-b con 0x7f9700141ae0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:42.067 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.067 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97000b48e0 con 0x7f970013ecc0 2023-10-30T17:31:42.067 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.068 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97000a8b60 con 0x7f97000b4500 2023-10-30T17:31:42.068 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.068 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97000aa5a0 con 0x7f9700141ae0 2023-10-30T17:31:42.068 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:42.068 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f9705141640 10 monclient(hunting): authenticate will time out at 1275.419417s 2023-10-30T17:31:42.069 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.058+0000 7f96fed9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f97000b4500 0x7f97000a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.069 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fe59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f970013ecc0 0x7f970013f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.069 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96ff59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f9700141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.069 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fed9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f97000b4500 0x7f97000a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:43042/0 (socket says 172.21.15.177:43042) 2023-10-30T17:31:42.070 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fed9d640 1 -- 172.21.15.177:0/4149501608 learned_addr learned my addr 172.21.15.177:0/4149501608 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:42.070 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fed9d640 10 monclient(hunting): get_auth_request con 0x7f97000b4500 auth_method 0 2023-10-30T17:31:42.070 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.070 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.070 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.071 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96ff59e640 10 monclient(hunting): get_auth_request con 0x7f9700141ae0 auth_method 0 2023-10-30T17:31:42.071 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96ff59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.071 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96ff59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.071 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96ff59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.071 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fe59c640 10 monclient(hunting): get_auth_request con 0x7f970013ecc0 auth_method 0 2023-10-30T17:31:42.072 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.072 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fe59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.072 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fe59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.072 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.072 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.073 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.059+0000 7f96fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:42.073 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.060+0000 7f96ff59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.073 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.060+0000 7f96ff59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.073 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.060+0000 7f96ff59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:42.073 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.060+0000 7f96fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.074 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.060+0000 7f96fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.074 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.060+0000 7f96fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:42.074 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.061+0000 7f96ff59e640 10 monclient(hunting): handle_auth_done global_id 4124 payload 274 2023-10-30T17:31:42.074 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.061+0000 7f96ff59e640 1 -- 172.21.15.177:0/4149501608 >> v2:172.21.15.177:3300/0 conn(0x7f97000b4500 msgr2=0x7f97000a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:42.074 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.061+0000 7f96ff59e640 1 --2- 172.21.15.177:0/4149501608 >> v2:172.21.15.177:3300/0 conn(0x7f97000b4500 0x7f97000a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.075 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.061+0000 7f96ff59e640 1 -- 172.21.15.177:0/4149501608 >> v2:172.21.15.170:3300/0 conn(0x7f970013ecc0 msgr2=0x7f970013f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:42.075 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.061+0000 7f96ff59e640 1 --2- 172.21.15.177:0/4149501608 >> v2:172.21.15.170:3300/0 conn(0x7f970013ecc0 0x7f970013f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.075 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.061+0000 7f96ff59e640 10 monclient: _finish_hunting 0 2023-10-30T17:31:42.075 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.061+0000 7f96ff59e640 1 monclient: found mon.noname-b 2023-10-30T17:31:42.075 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.061+0000 7f96ff59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:42.076 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.061+0000 7f96ff59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:31:42.076 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.061+0000 7f96ff59e640 1 -- 172.21.15.177:0/4149501608 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97000a8ff0 con 0x7f9700141ae0 2023-10-30T17:31:42.076 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.063+0000 7f96ff59e640 1 --2- 172.21.15.177:0/4149501608 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f9700141f50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f97000b4b50 tx=0x7f96f802bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fcecdafc50a5038 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:42.080 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.065+0000 7f96fdd9b640 1 -- 172.21.15.177:0/4149501608 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f802cb40 con 0x7f9700141ae0 2023-10-30T17:31:42.081 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.065+0000 7f96fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:42.081 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.065+0000 7f96fdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:31:42.081 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.065+0000 7f96fdd9b640 10 monclient: dump: 2023-10-30T17:31:42.081 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:42.082 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.082 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.082 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.082 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:42.082 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.083 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:42.083 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:42.083 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:42.083 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.083 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.065+0000 7f96fdd9b640 10 monclient: _finish_auth 0 2023-10-30T17:31:42.083 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.065+0000 7f96fdd9b640 10 monclient: _check_auth_tickets 2023-10-30T17:31:42.084 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.065+0000 7f96fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:42.084 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.065+0000 7f9705141640 5 monclient: authenticate success, global_id 4124 2023-10-30T17:31:42.084 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.065+0000 7f9705141640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:31:42.084 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.067+0000 7f96fdd9b640 1 -- 172.21.15.177:0/4149501608 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f802cca0 con 0x7f9700141ae0 2023-10-30T17:31:42.084 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.067+0000 7f96fdd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:42.085 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.067+0000 7f96fdd9b640 1 -- 172.21.15.177:0/4149501608 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f8034660 con 0x7f9700141ae0 2023-10-30T17:31:42.085 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.067+0000 7f96fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:42.085 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.067+0000 7f96fdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:31:42.085 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.067+0000 7f96fdd9b640 10 monclient: dump: 2023-10-30T17:31:42.085 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:42.085 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.086 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.086 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.086 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:42.086 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.086 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:42.087 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:42.087 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:42.087 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.087 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.067+0000 7f9705141640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:42.087 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.067+0000 7f9705141640 4 set_mon_vals no callback set 2023-10-30T17:31:42.087 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 10 monclient: shutdown 2023-10-30T17:31:42.088 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:42.088 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 1 -- 172.21.15.177:0/4149501608 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 msgr2=0x7f9700141f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:42.088 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 1 --2- 172.21.15.177:0/4149501608 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f9700141f50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f97000b4b50 tx=0x7f96f802bd90 comp rx=0 tx=0).stop 2023-10-30T17:31:42.088 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 1 -- 172.21.15.177:0/4149501608 shutdown_connections 2023-10-30T17:31:42.088 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 1 --2- 172.21.15.177:0/4149501608 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f9700141f50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.089 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 1 --2- 172.21.15.177:0/4149501608 >> v2:172.21.15.170:3300/0 conn(0x7f970013ecc0 0x7f970013f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.089 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 1 --2- 172.21.15.177:0/4149501608 >> v2:172.21.15.177:3300/0 conn(0x7f97000b4500 0x7f97000a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.089 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 1 -- 172.21.15.177:0/4149501608 >> 172.21.15.177:0/4149501608 conn(0x7f9700071b60 msgr2=0x7f9700071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:42.090 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 1 -- 172.21.15.177:0/4149501608 shutdown_connections 2023-10-30T17:31:42.090 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.068+0000 7f9705141640 1 -- 172.21.15.177:0/4149501608 wait complete. 2023-10-30T17:31:42.090 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.069+0000 7f9705141640 10 monclient: build_initial_monmap 2023-10-30T17:31:42.090 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.069+0000 7f9705141640 10 monclient: monmap: 2023-10-30T17:31:42.091 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:42.091 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:42.091 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2023-10-30T17:31:42.091 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2023-10-30T17:31:42.091 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:42.091 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.092 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:42.092 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:42.092 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:42.092 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.092 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.069+0000 7f9705141640 1 librados: starting msgr at 2023-10-30T17:31:42.092 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.069+0000 7f9705141640 1 librados: starting objecter 2023-10-30T17:31:42.093 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.069+0000 7f9705141640 1 Processor -- start 2023-10-30T17:31:42.093 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.069+0000 7f9705141640 1 -- start start 2023-10-30T17:31:42.093 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.069+0000 7f9705141640 1 librados: setting wanted keys 2023-10-30T17:31:42.093 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.069+0000 7f9705141640 1 librados: calling monclient init 2023-10-30T17:31:42.093 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.069+0000 7f9705141640 10 monclient: init 2023-10-30T17:31:42.094 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:42.094 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:31:42.094 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97000b4500 0x7f97001cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.094 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 10 monclient(hunting): picked mon.noname-b con 0x7f97000b4500 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:42.094 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f970013ecc0 0x7f97000b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.095 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 10 monclient(hunting): picked mon.noname-a con 0x7f970013ecc0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:42.095 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f97000b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.095 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 10 monclient(hunting): picked mon.noname-c con 0x7f9700141ae0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:42.095 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.095 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97000b7fc0 con 0x7f970013ecc0 2023-10-30T17:31:42.096 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.096 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97001d6170 con 0x7f97000b4500 2023-10-30T17:31:42.096 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.096 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97001d62e0 con 0x7f9700141ae0 2023-10-30T17:31:42.096 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:42.096 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.070+0000 7f9705141640 10 monclient(hunting): authenticate will time out at 1275.431500s 2023-10-30T17:31:42.097 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96ff59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f97000b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.097 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96fe59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f970013ecc0 0x7f97000b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.097 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96fed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97000b4500 0x7f97001cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.097 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96ff59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f97000b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.177:50848/0 (socket says 172.21.15.177:50848) 2023-10-30T17:31:42.097 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96ff59e640 1 -- 172.21.15.177:0/148983917 learned_addr learned my addr 172.21.15.177:0/148983917 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:42.098 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96ff59e640 10 monclient(hunting): get_auth_request con 0x7f9700141ae0 auth_method 0 2023-10-30T17:31:42.098 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96ff59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.098 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96ff59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.098 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96ff59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.098 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96fed9d640 10 monclient(hunting): get_auth_request con 0x7f97000b4500 auth_method 0 2023-10-30T17:31:42.099 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.099 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96fed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.099 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96fed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.099 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96fe59c640 10 monclient(hunting): get_auth_request con 0x7f970013ecc0 auth_method 0 2023-10-30T17:31:42.099 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.100 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96fe59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.100 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96fe59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.100 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96ff59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.100 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96ff59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.100 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.071+0000 7f96ff59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:42.101 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.072+0000 7f96fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.101 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.072+0000 7f96fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.101 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.072+0000 7f96fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:42.101 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.072+0000 7f96fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.101 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.072+0000 7f96fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.102 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.072+0000 7f96fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:42.102 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.074+0000 7f96ff59e640 10 monclient(hunting): handle_auth_done global_id 4127 payload 824 2023-10-30T17:31:42.102 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.074+0000 7f96ff59e640 1 -- 172.21.15.177:0/148983917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97000b4500 msgr2=0x7f97001cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:42.102 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.074+0000 7f96ff59e640 1 --2- 172.21.15.177:0/148983917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97000b4500 0x7f97001cb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.102 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.074+0000 7f96ff59e640 1 -- 172.21.15.177:0/148983917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f970013ecc0 msgr2=0x7f97000b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:42.103 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.074+0000 7f96ff59e640 1 --2- 172.21.15.177:0/148983917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f970013ecc0 0x7f97000b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.103 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.074+0000 7f96ff59e640 10 monclient: _finish_hunting 0 2023-10-30T17:31:42.103 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.074+0000 7f96ff59e640 1 monclient: found mon.noname-c 2023-10-30T17:31:42.103 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.074+0000 7f96ff59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:42.103 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.074+0000 7f96ff59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:31:42.104 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.074+0000 7f96ff59e640 1 -- 172.21.15.177:0/148983917 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f8009a70 con 0x7f9700141ae0 2023-10-30T17:31:42.104 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.076+0000 7f96ff59e640 1 --2- 172.21.15.177:0/148983917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f97000b5300 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009e80 tx=0x7f96f8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:42.104 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.076+0000 7f96e77fe640 10 client.?.objecter ms_handle_connect 0x7f9700141ae0 2023-10-30T17:31:42.104 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.076+0000 7f96e77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:42.106 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 1 -- 172.21.15.177:0/148983917 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f8040650 con 0x7f9700141ae0 2023-10-30T17:31:42.106 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:42.107 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:31:42.107 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 10 monclient: dump: 2023-10-30T17:31:42.107 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:42.107 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.107 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.108 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.108 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:42.108 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.108 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:42.108 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:42.108 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:42.109 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.109 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 10 monclient: _finish_auth 0 2023-10-30T17:31:42.109 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 10 monclient: _check_auth_tickets 2023-10-30T17:31:42.109 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:42.109 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 1 -- 172.21.15.177:0/148983917 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f80407b0 con 0x7f9700141ae0 2023-10-30T17:31:42.110 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f9705141640 5 monclient: authenticate success, global_id 4127 2023-10-30T17:31:42.110 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f9705141640 10 monclient: _renew_subs 2023-10-30T17:31:42.110 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f9705141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:31:42.111 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f9705141640 1 -- 172.21.15.177:0/148983917 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97000b7be0 con 0x7f9700141ae0 2023-10-30T17:31:42.111 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:42.111 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 1 -- 172.21.15.177:0/148983917 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f8040b60 con 0x7f9700141ae0 2023-10-30T17:31:42.111 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:42.111 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:31:42.112 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96e77fe640 10 monclient: dump: 2023-10-30T17:31:42.112 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:42.112 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.112 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.112 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.113 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:42.113 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.113 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:42.113 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:42.113 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:42.114 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.114 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f9705141640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:42.114 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f9705141640 10 monclient: _renew_subs 2023-10-30T17:31:42.114 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f96fd59a640 4 set_mon_vals no callback set 2023-10-30T17:31:42.114 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f9705141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:31:42.115 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f9705141640 1 -- 172.21.15.177:0/148983917 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97000b60d0 con 0x7f9700141ae0 2023-10-30T17:31:42.115 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.080+0000 7f9705141640 1 librados: init done 2023-10-30T17:31:42.115 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.081+0000 7f9705141640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:42.115 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.081+0000 7f9705141640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:42.115 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.081+0000 7f9705141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:31:42.115 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.081+0000 7f9705141640 1 -- 172.21.15.177:0/148983917 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97000b8190 con 0x7f9700141ae0 2023-10-30T17:31:42.116 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.100+0000 7f96e77fe640 1 -- 172.21.15.177:0/148983917 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f96f8052430 con 0x7f9700141ae0 2023-10-30T17:31:42.116 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.101+0000 7f96e77fe640 1 -- 172.21.15.177:0/148983917 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f96f803e890 con 0x7f9700141ae0 2023-10-30T17:31:42.116 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.101+0000 7f96e77fe640 10 client.4127.objecter ms_dispatch 0x7f9700000b90 osd_map(5..5 src has 1..5) v4 2023-10-30T17:31:42.116 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.101+0000 7f96e77fe640 3 client.4127.objecter handle_osd_map got epochs [5,5] > 0 2023-10-30T17:31:42.116 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.101+0000 7f96e77fe640 3 client.4127.objecter handle_osd_map decoding full epoch 5 2023-10-30T17:31:42.117 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.101+0000 7f96e77fe640 20 client.4127.objecter dump_active .. 0 homeless 2023-10-30T17:31:42.117 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:42.112+0000 7f2fb4c2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-30T17:31:42.161 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:42.160+0000 7f2b9a22b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-30T17:31:42.250 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:42.249+0000 7f2fb4c2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T17:31:42.277 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.276+0000 7f96e77fe640 1 -- 172.21.15.177:0/148983917 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f97000b8190 con 0x7f9700141ae0 2023-10-30T17:31:42.277 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.276+0000 7f96e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:42.277 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.276+0000 7f96e77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:42.296 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:42.295+0000 7f2b9a22b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T17:31:42.348 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.347+0000 7f9705141640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "46c2dd9a-c073-42ab-a07b-918f83b8f7f6", "id": 3}] 2023-10-30T17:31:42.349 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.347+0000 7f9705141640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "46c2dd9a-c073-42ab-a07b-918f83b8f7f6", "id": 3}] 2023-10-30T17:31:42.349 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.347+0000 7f9705141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:31:42.349 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.347+0000 7f9705141640 1 -- 172.21.15.177:0/148983917 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd new", "uuid": "46c2dd9a-c073-42ab-a07b-918f83b8f7f6", "id": 3} v 0) v1 -- 0x7f9700067cd0 con 0x7f9700141ae0 2023-10-30T17:31:42.366 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:42.365+0000 7f2fb4c2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T17:31:42.404 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:42.402+0000 7f2b9a22b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T17:31:42.416 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:42.415+0000 7f2fb4c2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T17:31:42.452 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:42.450+0000 7f2b9a22b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T17:31:42.470 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.469+0000 7f96e77fe640 1 -- 172.21.15.177:0/148983917 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "46c2dd9a-c073-42ab-a07b-918f83b8f7f6", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f96f8044020 con 0x7f9700141ae0 2023-10-30T17:31:42.471 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.469+0000 7f96e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "46c2dd9a-c073-42ab-a07b-918f83b8f7f6", "id": 3}] 2023-10-30T17:31:42.471 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.469+0000 7f96e77fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:42.472 INFO:teuthology.orchestra.run.smithi177.stdout:3 2023-10-30T17:31:42.474 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 10 librados: watch_flush enter 2023-10-30T17:31:42.474 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 10 librados: watch_flush exit 2023-10-30T17:31:42.474 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 20 client.4127.objecter shutdown clearing up homeless session... 2023-10-30T17:31:42.474 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 10 client.4127.objecter successfully canceled tick 2023-10-30T17:31:42.475 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 10 monclient: shutdown 2023-10-30T17:31:42.475 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:42.475 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 1 -- 172.21.15.177:0/148983917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 msgr2=0x7f97000b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:42.475 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 1 --2- 172.21.15.177:0/148983917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f97000b5300 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009e80 tx=0x7f96f8004a40 comp rx=0 tx=0).stop 2023-10-30T17:31:42.475 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 1 -- 172.21.15.177:0/148983917 shutdown_connections 2023-10-30T17:31:42.476 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 1 --2- 172.21.15.177:0/148983917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9700141ae0 0x7f97000b5300 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.476 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 1 --2- 172.21.15.177:0/148983917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f970013ecc0 0x7f97000b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.476 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 1 --2- 172.21.15.177:0/148983917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97000b4500 0x7f97001cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.476 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 1 -- 172.21.15.177:0/148983917 >> 172.21.15.177:0/148983917 conn(0x7f9700071b60 msgr2=0x7f97000b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:42.476 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 1 -- 172.21.15.177:0/148983917 shutdown_connections 2023-10-30T17:31:42.477 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 1 -- 172.21.15.177:0/148983917 wait complete. 2023-10-30T17:31:42.477 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.470+0000 7f9705141640 1 librados: shutdown 2023-10-30T17:31:42.482 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 94b9a549-2c12-42bb-b3f1-2e3304c574e1 4 2023-10-30T17:31:42.519 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:42.518+0000 7f2fb4c2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T17:31:42.553 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:42.552+0000 7f2b9a22b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T17:31:42.581 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.579+0000 7f4da04e0640 10 monclient: get_monmap_and_config 2023-10-30T17:31:42.581 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.579+0000 7f4da04e0640 10 monclient: build_initial_monmap 2023-10-30T17:31:42.581 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.579+0000 7f4da04e0640 10 monclient: monmap: 2023-10-30T17:31:42.582 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:42.582 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.582 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:31:42.580923+0000 2023-10-30T17:31:42.582 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:31:42.580923+0000 2023-10-30T17:31:42.582 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:42.583 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.583 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:42.583 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:42.583 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:42.583 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:42.583 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:42.584 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.587 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.580+0000 7f4da04e0640 1 Processor -- start 2023-10-30T17:31:42.587 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.580+0000 7f4da04e0640 1 -- start start 2023-10-30T17:31:42.587 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.580+0000 7f4da04e0640 10 monclient: init 2023-10-30T17:31:42.588 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:42.588 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T17:31:42.588 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4d980b4500 0x7f4d980a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.588 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 10 monclient(hunting): picked mon.noname-a con 0x7f4d980b4500 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:42.589 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d9813ecc0 addr v1:172.21.15.170:6789/0 2023-10-30T17:31:42.589 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d98141ae0 addr v1:172.21.15.177:6789/0 2023-10-30T17:31:42.589 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d980a8b60 con 0x7f4d9813ecc0 2023-10-30T17:31:42.589 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d980aa5a0 con 0x7f4d98141ae0 2023-10-30T17:31:42.590 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.590 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d980b48e0 con 0x7f4d980b4500 2023-10-30T17:31:42.590 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:42.591 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4da04e0640 10 monclient(hunting): authenticate will time out at 1275.942357s 2023-10-30T17:31:42.591 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4d9ea7d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f4d98141ae0 0x7f4d98141f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:38252/0 (socket says 172.21.15.177:38252) 2023-10-30T17:31:42.591 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.581+0000 7f4d9ea7d640 1 -- 172.21.15.177:0/2032539443 learned_addr learned my addr 172.21.15.177:0/2032539443 (peer_addr_for_me v1:172.21.15.177:0/0) 2023-10-30T17:31:42.592 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.582+0000 7f4d9e27c640 1 --2- 172.21.15.177:0/2032539443 >> v2:172.21.15.170:3300/0 conn(0x7f4d980b4500 0x7f4d980a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.592 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.583+0000 7f4d9e27c640 10 monclient(hunting): get_auth_request con 0x7f4d980b4500 auth_method 0 2023-10-30T17:31:42.592 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.583+0000 7f4d9e27c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.592 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.583+0000 7f4d9e27c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.593 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.583+0000 7f4d9e27c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.593 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.584+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75741983 0 0) 0x7f4d980aa5a0 con 0x7f4d98141ae0 2023-10-30T17:31:42.593 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.584+0000 7f4d9d27a640 10 monclient(hunting): _negotiate 2023-10-30T17:31:42.593 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.584+0000 7f4d9d27a640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.594 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.584+0000 7f4d9d27a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.594 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.584+0000 7f4d9d27a640 10 monclient(hunting): my global_id is 4135 2023-10-30T17:31:42.594 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.584+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d84003520 con 0x7f4d98141ae0 2023-10-30T17:31:42.595 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.585+0000 7f4d9e27c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.595 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.585+0000 7f4d9e27c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.595 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.585+0000 7f4d9e27c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:42.596 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.586+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838916152 0 0) 0x7f4d980a8b60 con 0x7f4d9813ecc0 2023-10-30T17:31:42.596 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.586+0000 7f4d9d27a640 10 monclient(hunting): _negotiate 2023-10-30T17:31:42.596 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.586+0000 7f4d9d27a640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.596 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.586+0000 7f4d9d27a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.597 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.586+0000 7f4d9d27a640 10 monclient(hunting): my global_id is 4140 2023-10-30T17:31:42.597 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.586+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d980aa5a0 con 0x7f4d9813ecc0 2023-10-30T17:31:42.605 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2842055162 0 0) 0x7f4d84003520 con 0x7f4d98141ae0 2023-10-30T17:31:42.605 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 >> v2:172.21.15.170:3300/0 conn(0x7f4d980b4500 msgr2=0x7f4d980a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:42.605 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 --2- 172.21.15.177:0/2032539443 >> v2:172.21.15.170:3300/0 conn(0x7f4d980b4500 0x7f4d980a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.605 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 >> v1:172.21.15.170:6789/0 conn(0x7f4d9813ecc0 legacy=0x7f4d9813f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:42.606 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient: _finish_hunting 0 2023-10-30T17:31:42.606 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 monclient: found mon.noname-c-legacy 2023-10-30T17:31:42.606 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:42.606 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2023-10-30T17:31:42.606 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d980a8ff0 con 0x7f4d98141ae0 2023-10-30T17:31:42.607 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient: _finish_auth 0 2023-10-30T17:31:42.607 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient: _check_auth_tickets 2023-10-30T17:31:42.607 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:42.607 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4da04e0640 5 monclient: authenticate success, global_id 4135 2023-10-30T17:31:42.607 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4da04e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:31:42.608 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2465063691 0 0) 0x7f4d900019f0 con 0x7f4d98141ae0 2023-10-30T17:31:42.608 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:42.608 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T17:31:42.608 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient: dump: 2023-10-30T17:31:42.608 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:42.609 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.609 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.609 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.609 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:42.609 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.610 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:42.610 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:42.610 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:42.610 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.610 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2023-10-30T17:31:42.611 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:42.611 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 >> v1:172.21.15.177:6789/0 conn(0x7f4d98141ae0 legacy=0x7f4d98141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:42.611 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:31:42.611 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 --2- 172.21.15.177:0/2032539443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d84003b50 0x7f4d84003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.611 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient(hunting): picked mon.a con 0x7f4d84003b50 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:42.612 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 --2- 172.21.15.177:0/2032539443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d840047c0 0x7f4d84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.612 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient(hunting): picked mon.b con 0x7f4d840047c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:42.612 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 1 --2- 172.21.15.177:0/2032539443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d840051d0 0x7f4d8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.612 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.588+0000 7f4d9d27a640 10 monclient(hunting): picked mon.c con 0x7f4d840051d0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:42.612 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.589+0000 7f4d9d27a640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.613 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.589+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d980b48e0 con 0x7f4d84003b50 2023-10-30T17:31:42.613 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.589+0000 7f4d9d27a640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.613 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.589+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8400c100 con 0x7f4d840047c0 2023-10-30T17:31:42.613 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.589+0000 7f4d9d27a640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.613 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.589+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8400c2c0 con 0x7f4d840051d0 2023-10-30T17:31:42.614 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.589+0000 7f4d9d27a640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:42.614 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.589+0000 7f4d9d27a640 10 monclient(hunting): _finish_auth 0 2023-10-30T17:31:42.614 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.589+0000 7f4da04e0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T17:31:42.614 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.589+0000 7f4d9ea7d640 1 --2- 172.21.15.177:0/2032539443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d840051d0 0x7f4d8400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.614 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.590+0000 7f4d9ea7d640 10 monclient(hunting): get_auth_request con 0x7f4d840051d0 auth_method 0 2023-10-30T17:31:42.615 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.590+0000 7f4d9ea7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.615 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.590+0000 7f4d9ea7d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.615 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.590+0000 7f4d9ea7d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:31:42.615 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.590+0000 7f4d9ea7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.615 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.590+0000 7f4d9ea7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.616 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.590+0000 7f4d9ea7d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T17:31:42.616 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 10 monclient(hunting): handle_auth_done global_id 4135 payload 293 2023-10-30T17:31:42.616 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9da7b640 1 --2- 172.21.15.177:0/2032539443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d840047c0 0x7f4d84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.616 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 1 -- 172.21.15.177:0/2032539443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d840047c0 msgr2=0x7f4d84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:42.617 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 1 --2- 172.21.15.177:0/2032539443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d840047c0 0x7f4d84004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.617 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 1 -- 172.21.15.177:0/2032539443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d84003b50 msgr2=0x7f4d84003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:42.617 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 1 --2- 172.21.15.177:0/2032539443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d84003b50 0x7f4d84003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.617 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 10 monclient: _finish_hunting 0 2023-10-30T17:31:42.618 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 1 monclient: found mon.c 2023-10-30T17:31:42.618 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:42.618 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:31:42.618 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 1 -- 172.21.15.177:0/2032539443 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d84003520 con 0x7f4d840051d0 2023-10-30T17:31:42.619 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 10 monclient: _finish_auth 0 2023-10-30T17:31:42.619 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 10 monclient: _check_auth_tickets 2023-10-30T17:31:42.619 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.592+0000 7f4d9ea7d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:42.619 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.593+0000 7f4d9ea7d640 1 -- 172.21.15.177:0/2032539443 reap_dead start 2023-10-30T17:31:42.621 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.594+0000 7f4d9ea7d640 1 --2- 172.21.15.177:0/2032539443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d840051d0 0x7f4d8400b980 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000a710 tx=0x7f4d840057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d77733e94808eb80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:42.621 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.598+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d90034630 con 0x7f4d840051d0 2023-10-30T17:31:42.621 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.598+0000 7f4d9d27a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:42.622 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.598+0000 7f4d9d27a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:31:42.622 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.598+0000 7f4d9d27a640 10 monclient: dump: 2023-10-30T17:31:42.622 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:42.622 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.622 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.623 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.623 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:42.623 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.623 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:42.623 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:42.624 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:42.624 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.624 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.598+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9002f070 con 0x7f4d840051d0 2023-10-30T17:31:42.624 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.598+0000 7f4d9d27a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:42.625 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.598+0000 7f4d9d27a640 1 -- 172.21.15.177:0/2032539443 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d900349f0 con 0x7f4d840051d0 2023-10-30T17:31:42.625 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.598+0000 7f4d9d27a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:42.625 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.598+0000 7f4d9d27a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:31:42.625 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.598+0000 7f4d9d27a640 10 monclient: dump: 2023-10-30T17:31:42.626 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:42.626 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.626 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.626 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.627 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:42.627 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.627 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:42.627 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:42.628 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:42.628 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.628 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:42.628 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 4 set_mon_vals no callback set 2023-10-30T17:31:42.628 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 10 monclient: shutdown 2023-10-30T17:31:42.629 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:42.629 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 1 -- 172.21.15.177:0/2032539443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d840051d0 msgr2=0x7f4d8400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:42.629 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 1 --2- 172.21.15.177:0/2032539443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d840051d0 0x7f4d8400b980 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000a710 tx=0x7f4d840057f0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.629 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 1 -- 172.21.15.177:0/2032539443 shutdown_connections 2023-10-30T17:31:42.630 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 1 --2- 172.21.15.177:0/2032539443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d840051d0 0x7f4d8400b980 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.630 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 1 -- 172.21.15.177:0/2032539443 >> 172.21.15.177:0/2032539443 conn(0x7f4d98071b60 msgr2=0x7f4d98071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:42.630 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 1 -- 172.21.15.177:0/2032539443 shutdown_connections 2023-10-30T17:31:42.630 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.599+0000 7f4da04e0640 1 -- 172.21.15.177:0/2032539443 wait complete. 2023-10-30T17:31:42.631 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.600+0000 7f4da04e0640 10 monclient: build_initial_monmap 2023-10-30T17:31:42.631 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.600+0000 7f4da04e0640 10 monclient: monmap: 2023-10-30T17:31:42.631 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:42.631 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:42.632 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2023-10-30T17:31:42.632 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2023-10-30T17:31:42.632 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:42.632 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.633 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:42.633 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:42.633 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:42.633 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.634 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.600+0000 7f4da04e0640 1 librados: starting msgr at 2023-10-30T17:31:42.634 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.600+0000 7f4da04e0640 1 librados: starting objecter 2023-10-30T17:31:42.634 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.600+0000 7f4da04e0640 1 Processor -- start 2023-10-30T17:31:42.634 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.600+0000 7f4da04e0640 1 -- start start 2023-10-30T17:31:42.635 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.600+0000 7f4da04e0640 1 librados: setting wanted keys 2023-10-30T17:31:42.635 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.600+0000 7f4da04e0640 1 librados: calling monclient init 2023-10-30T17:31:42.635 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.600+0000 7f4da04e0640 10 monclient: init 2023-10-30T17:31:42.635 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:42.636 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:31:42.636 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d840051d0 0x7f4d981cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.636 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 10 monclient(hunting): picked mon.noname-a con 0x7f4d840051d0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:42.636 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d980b6b60 0x7f4d981da090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.637 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 10 monclient(hunting): picked mon.noname-b con 0x7f4d980b6b60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:42.637 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d980b58c0 0x7f4d981da5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.637 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 10 monclient(hunting): picked mon.noname-c con 0x7f4d980b58c0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:42.637 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.637 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d981ccd10 con 0x7f4d840051d0 2023-10-30T17:31:42.638 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.638 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d980b66d0 con 0x7f4d980b6b60 2023-10-30T17:31:42.638 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.638 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d980b62f0 con 0x7f4d980b58c0 2023-10-30T17:31:42.638 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:42.639 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4da04e0640 10 monclient(hunting): authenticate will time out at 1275.962458s 2023-10-30T17:31:42.639 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.601+0000 7f4d9da7b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d980b6b60 0x7f4d981da090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.639 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9e27c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d840051d0 0x7f4d981cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.639 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9ea7d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d980b58c0 0x7f4d981da5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.639 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9da7b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d980b6b60 0x7f4d981da090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:43074/0 (socket says 172.21.15.177:43074) 2023-10-30T17:31:42.640 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9e27c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d840051d0 0x7f4d981cc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.177:44364/0 (socket says 172.21.15.177:44364) 2023-10-30T17:31:42.640 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9da7b640 1 -- 172.21.15.177:0/361843360 learned_addr learned my addr 172.21.15.177:0/361843360 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:42.640 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9da7b640 10 monclient(hunting): get_auth_request con 0x7f4d980b6b60 auth_method 0 2023-10-30T17:31:42.640 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9da7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.640 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9da7b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.641 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9da7b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.641 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9e27c640 10 monclient(hunting): get_auth_request con 0x7f4d840051d0 auth_method 0 2023-10-30T17:31:42.641 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9e27c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.641 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9e27c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.641 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9e27c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.641 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9ea7d640 10 monclient(hunting): get_auth_request con 0x7f4d980b58c0 auth_method 0 2023-10-30T17:31:42.642 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9ea7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.642 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9ea7d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.642 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9ea7d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.642 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9e27c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.642 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9e27c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.643 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9e27c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:42.643 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9da7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.643 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9da7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.643 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9da7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:42.643 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9ea7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:42.644 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9ea7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:42.644 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.602+0000 7f4d9ea7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:42.645 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:42.622+0000 7f2fb4c2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T17:31:42.646 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.605+0000 7f4d9e27c640 10 monclient(hunting): handle_auth_done global_id 4143 payload 824 2023-10-30T17:31:42.647 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.605+0000 7f4d9e27c640 1 -- 172.21.15.177:0/361843360 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d980b58c0 msgr2=0x7f4d981da5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:42.647 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.605+0000 7f4d9e27c640 1 --2- 172.21.15.177:0/361843360 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d980b58c0 0x7f4d981da5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.647 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.605+0000 7f4d9e27c640 1 -- 172.21.15.177:0/361843360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d980b6b60 msgr2=0x7f4d981da090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:42.647 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.605+0000 7f4d9e27c640 1 --2- 172.21.15.177:0/361843360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d980b6b60 0x7f4d981da090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.648 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.605+0000 7f4d9e27c640 10 monclient: _finish_hunting 0 2023-10-30T17:31:42.648 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.605+0000 7f4d9e27c640 1 monclient: found mon.noname-a 2023-10-30T17:31:42.648 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.605+0000 7f4d9e27c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:42.648 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.605+0000 7f4d9e27c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:42.648 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.605+0000 7f4d9e27c640 1 -- 172.21.15.177:0/361843360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d98140a30 con 0x7f4d840051d0 2023-10-30T17:31:42.649 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.608+0000 7f4d9e27c640 1 --2- 172.21.15.177:0/361843360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d840051d0 0x7f4d981cc2b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400e820 tx=0x7f4d9400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:42.649 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.608+0000 7f4d8affd640 10 client.?.objecter ms_handle_connect 0x7f4d840051d0 2023-10-30T17:31:42.649 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.608+0000 7f4d8affd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:42.649 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 1 -- 172.21.15.177:0/361843360 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94004020 con 0x7f4d840051d0 2023-10-30T17:31:42.650 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:42.650 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:42.650 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 10 monclient: dump: 2023-10-30T17:31:42.651 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:42.651 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.651 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.651 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.652 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:42.652 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.652 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:42.653 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:42.653 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:42.653 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.653 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 10 monclient: _finish_auth 0 2023-10-30T17:31:42.654 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 10 monclient: _check_auth_tickets 2023-10-30T17:31:42.654 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:42.654 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 1 -- 172.21.15.177:0/361843360 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d94004180 con 0x7f4d840051d0 2023-10-30T17:31:42.654 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:42.655 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 1 -- 172.21.15.177:0/361843360 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94005990 con 0x7f4d840051d0 2023-10-30T17:31:42.655 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:42.655 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:42.655 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d8affd640 10 monclient: dump: 2023-10-30T17:31:42.656 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:42.656 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.656 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.657 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:42.657 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:42.657 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.657 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:42.658 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:42.658 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:42.658 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.659 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4da04e0640 5 monclient: authenticate success, global_id 4143 2023-10-30T17:31:42.659 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4da04e0640 10 monclient: _renew_subs 2023-10-30T17:31:42.659 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4da04e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:42.659 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4d9ca79640 4 set_mon_vals no callback set 2023-10-30T17:31:42.660 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4da04e0640 1 -- 172.21.15.177:0/361843360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d98140c80 con 0x7f4d840051d0 2023-10-30T17:31:42.660 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4da04e0640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:42.660 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4da04e0640 10 monclient: _renew_subs 2023-10-30T17:31:42.660 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4da04e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:42.661 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4da04e0640 1 -- 172.21.15.177:0/361843360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d98141140 con 0x7f4d840051d0 2023-10-30T17:31:42.661 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.612+0000 7f4da04e0640 1 librados: init done 2023-10-30T17:31:42.661 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.613+0000 7f4da04e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:42.661 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.613+0000 7f4da04e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:42.662 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.613+0000 7f4da04e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:42.662 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.613+0000 7f4da04e0640 1 -- 172.21.15.177:0/361843360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d980b64b0 con 0x7f4d840051d0 2023-10-30T17:31:42.662 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:42.652+0000 7f2b9a22b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T17:31:42.710 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.708+0000 7f4d8affd640 1 -- 172.21.15.177:0/361843360 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4d9400c6d0 con 0x7f4d840051d0 2023-10-30T17:31:42.717 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.715+0000 7f4d8affd640 1 -- 172.21.15.177:0/361843360 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f4d94003d70 con 0x7f4d840051d0 2023-10-30T17:31:42.717 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.715+0000 7f4d8affd640 10 client.4143.objecter ms_dispatch 0x7f4d98000b90 osd_map(6..6 src has 1..6) v4 2023-10-30T17:31:42.717 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.715+0000 7f4d8affd640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2023-10-30T17:31:42.717 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.715+0000 7f4d8affd640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2023-10-30T17:31:42.718 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.715+0000 7f4d8affd640 20 client.4143.objecter dump_active .. 0 homeless 2023-10-30T17:31:42.718 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.715+0000 7f4d8affd640 1 -- 172.21.15.177:0/361843360 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f4d94039b40 con 0x7f4d840051d0 2023-10-30T17:31:42.718 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.715+0000 7f4d8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:42.718 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.715+0000 7f4d8affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:42.769 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:42.768+0000 7f2fb4c2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T17:31:42.788 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.786+0000 7f4da04e0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "94b9a549-2c12-42bb-b3f1-2e3304c574e1", "id": 4}] 2023-10-30T17:31:42.788 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.786+0000 7f4da04e0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "94b9a549-2c12-42bb-b3f1-2e3304c574e1", "id": 4}] 2023-10-30T17:31:42.788 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.787+0000 7f4da04e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:42.789 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.787+0000 7f4da04e0640 1 -- 172.21.15.177:0/361843360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "94b9a549-2c12-42bb-b3f1-2e3304c574e1", "id": 4} v 0) v1 -- 0x7f4d98067cd0 con 0x7f4d840051d0 2023-10-30T17:31:42.795 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:42.794+0000 7f2b9a22b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T17:31:42.829 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:42.828+0000 7f2fb4c2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T17:31:42.841 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:42.840+0000 7f2b9a22b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T17:31:42.872 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.870+0000 7f4d8affd640 1 -- 172.21.15.177:0/361843360 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "94b9a549-2c12-42bb-b3f1-2e3304c574e1", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4d94028030 con 0x7f4d840051d0 2023-10-30T17:31:42.872 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.870+0000 7f4d8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "94b9a549-2c12-42bb-b3f1-2e3304c574e1", "id": 4}] 2023-10-30T17:31:42.872 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.870+0000 7f4d8affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:42.873 INFO:teuthology.orchestra.run.smithi177.stdout:4 2023-10-30T17:31:42.874 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.871+0000 7f4da04e0640 10 librados: watch_flush enter 2023-10-30T17:31:42.874 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.871+0000 7f4da04e0640 10 librados: watch_flush exit 2023-10-30T17:31:42.874 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.871+0000 7f4da04e0640 20 client.4143.objecter shutdown clearing up homeless session... 2023-10-30T17:31:42.875 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.871+0000 7f4da04e0640 10 client.4143.objecter successfully canceled tick 2023-10-30T17:31:42.875 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 10 monclient: shutdown 2023-10-30T17:31:42.875 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:42.875 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 1 -- 172.21.15.177:0/361843360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d840051d0 msgr2=0x7f4d981cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:42.875 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 1 --2- 172.21.15.177:0/361843360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d840051d0 0x7f4d981cc2b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400e820 tx=0x7f4d9400ece0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.876 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 1 -- 172.21.15.177:0/361843360 shutdown_connections 2023-10-30T17:31:42.876 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 1 --2- 172.21.15.177:0/361843360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d980b6b60 0x7f4d981da090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.876 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 1 --2- 172.21.15.177:0/361843360 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d980b58c0 0x7f4d981da5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.876 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 1 --2- 172.21.15.177:0/361843360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d840051d0 0x7f4d981cc2b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:42.876 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 1 -- 172.21.15.177:0/361843360 >> 172.21.15.177:0/361843360 conn(0x7f4d98071b60 msgr2=0x7f4d9813da00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:42.877 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 1 -- 172.21.15.177:0/361843360 shutdown_connections 2023-10-30T17:31:42.877 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 1 -- 172.21.15.177:0/361843360 wait complete. 2023-10-30T17:31:42.877 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.872+0000 7f4da04e0640 1 librados: shutdown 2023-10-30T17:31:42.887 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 2314fc35-d5d3-4609-853a-c0cbf5212ea7 5 2023-10-30T17:31:42.986 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.984+0000 7fa9f1826640 10 monclient: get_monmap_and_config 2023-10-30T17:31:42.986 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.984+0000 7fa9f1826640 10 monclient: build_initial_monmap 2023-10-30T17:31:42.987 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.984+0000 7fa9f1826640 10 monclient: monmap: 2023-10-30T17:31:42.987 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:42.987 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:42.987 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:31:42.985948+0000 2023-10-30T17:31:42.988 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:31:42.985948+0000 2023-10-30T17:31:42.988 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:42.988 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:42.988 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:42.988 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:42.989 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:42.989 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:42.989 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:42.989 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:42.993 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.985+0000 7fa9f1826640 1 Processor -- start 2023-10-30T17:31:42.993 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.985+0000 7fa9f1826640 1 -- start start 2023-10-30T17:31:42.993 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.985+0000 7fa9f1826640 10 monclient: init 2023-10-30T17:31:42.994 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:42.994 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-10-30T17:31:42.994 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9ec13be60 0x7fa9ec13e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.994 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 10 monclient(hunting): picked mon.noname-b con 0x7fa9ec13be60 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:42.995 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa9ec072060 0x7fa9ec13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:42.995 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 10 monclient(hunting): picked mon.noname-c con 0x7fa9ec072060 addr v2:172.21.15.177:3300/0 2023-10-30T17:31:42.995 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9ec06ec70 addr v1:172.21.15.170:6789/0 2023-10-30T17:31:42.995 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ec071410 con 0x7fa9ec06ec70 2023-10-30T17:31:42.995 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.996 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ec0ae220 con 0x7fa9ec072060 2023-10-30T17:31:42.996 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:42.996 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec06e4d0 con 0x7fa9ec13be60 2023-10-30T17:31:42.996 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:42.997 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9f1826640 10 monclient(hunting): authenticate will time out at 1276.347381s 2023-10-30T17:31:42.997 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.986+0000 7fa9e3fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa9ec072060 0x7fa9ec13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.997 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9eb7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fa9ec06ec70 0x7fa9ec141cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.177:47198/0 (socket says 172.21.15.177:47198) 2023-10-30T17:31:42.997 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9eaffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9ec13be60 0x7fa9ec13e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:42.998 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9eb7fe640 1 -- 172.21.15.177:0/3805808259 learned_addr learned my addr 172.21.15.177:0/3805808259 (peer_addr_for_me v1:172.21.15.177:0/0) 2023-10-30T17:31:42.998 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9e3fff640 10 monclient(hunting): get_auth_request con 0x7fa9ec072060 auth_method 0 2023-10-30T17:31:42.998 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:42.998 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9e3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:42.999 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:42.999 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9eaffd640 10 monclient(hunting): get_auth_request con 0x7fa9ec13be60 auth_method 0 2023-10-30T17:31:42.999 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.000 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9eaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.000 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.000 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.000 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.000 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.987+0000 7fa9e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.001 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.988+0000 7fa9eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.001 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.988+0000 7fa9eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.001 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.988+0000 7fa9eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.001 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9e3fff640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2023-10-30T17:31:43.001 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9e3fff640 1 -- 172.21.15.177:0/3805808259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9ec13be60 msgr2=0x7fa9ec13e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:43.002 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9e3fff640 1 --2- 172.21.15.177:0/3805808259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9ec13be60 0x7fa9ec13e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.002 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9e3fff640 1 -- 172.21.15.177:0/3805808259 >> v1:172.21.15.170:6789/0 conn(0x7fa9ec06ec70 legacy=0x7fa9ec141cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.002 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9e3fff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:43.002 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9e3fff640 1 monclient: found mon.noname-c 2023-10-30T17:31:43.003 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9ea7fc640 1 -- 172.21.15.177:0/3805808259 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113395938 0 0) 0x7fa9ec071410 con 0x7fa9ec06ec70 2023-10-30T17:31:43.003 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:43.003 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9e3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:31:43.003 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9e3fff640 1 -- 172.21.15.177:0/3805808259 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ec06e610 con 0x7fa9ec072060 2023-10-30T17:31:43.003 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.989+0000 7fa9ea7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T17:31:43.004 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.991+0000 7fa9e3fff640 1 --2- 172.21.15.177:0/3805808259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 0x7fa9ec13ecf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40097a0 tx=0x7fa9d402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a31b421b18b7a14d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:43.004 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:42.991+0000 7f2fb4c2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T17:31:43.005 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:42.993+0000 7f2b9a22b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T17:31:43.011 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 1 -- 172.21.15.177:0/3805808259 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d402e070 con 0x7fa9ec072060 2023-10-30T17:31:43.011 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.012 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:31:43.012 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 10 monclient: dump: 2023-10-30T17:31:43.012 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.012 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.013 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.013 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.013 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.013 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.013 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.014 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.014 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.014 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.014 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 10 monclient: _finish_auth 0 2023-10-30T17:31:43.014 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 10 monclient: _check_auth_tickets 2023-10-30T17:31:43.015 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:43.015 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 1 -- 172.21.15.177:0/3805808259 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d40315a0 con 0x7fa9ec072060 2023-10-30T17:31:43.015 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:43.015 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 1 -- 172.21.15.177:0/3805808259 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4031930 con 0x7fa9ec072060 2023-10-30T17:31:43.016 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9f1826640 5 monclient: authenticate success, global_id 4141 2023-10-30T17:31:43.016 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9f1826640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:43.016 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9f1826640 4 set_mon_vals no callback set 2023-10-30T17:31:43.016 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.016 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:31:43.017 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9ea7fc640 10 monclient: dump: 2023-10-30T17:31:43.017 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.017 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.017 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.018 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.018 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.018 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.018 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.019 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.019 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.019 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.020 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9f1826640 10 monclient: shutdown 2023-10-30T17:31:43.020 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9f1826640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:43.020 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9f1826640 1 -- 172.21.15.177:0/3805808259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 msgr2=0x7fa9ec13ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.020 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9f1826640 1 --2- 172.21.15.177:0/3805808259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 0x7fa9ec13ecf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40097a0 tx=0x7fa9d402b840 comp rx=0 tx=0).stop 2023-10-30T17:31:43.020 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.995+0000 7fa9f1826640 1 -- 172.21.15.177:0/3805808259 shutdown_connections 2023-10-30T17:31:43.021 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.996+0000 7fa9f1826640 1 --2- 172.21.15.177:0/3805808259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 0x7fa9ec13ecf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.021 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.996+0000 7fa9f1826640 1 --2- 172.21.15.177:0/3805808259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9ec13be60 0x7fa9ec13e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.021 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.996+0000 7fa9f1826640 1 -- 172.21.15.177:0/3805808259 >> 172.21.15.177:0/3805808259 conn(0x7fa9ec0b4020 msgr2=0x7fa9ec0b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:43.021 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.996+0000 7fa9f1826640 1 -- 172.21.15.177:0/3805808259 shutdown_connections 2023-10-30T17:31:43.022 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.996+0000 7fa9f1826640 1 -- 172.21.15.177:0/3805808259 wait complete. 2023-10-30T17:31:43.022 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.996+0000 7fa9f1826640 10 monclient: build_initial_monmap 2023-10-30T17:31:43.022 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.996+0000 7fa9f1826640 10 monclient: monmap: 2023-10-30T17:31:43.022 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:43.023 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:43.023 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2023-10-30T17:31:43.023 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2023-10-30T17:31:43.023 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:43.024 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.024 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:43.024 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:43.024 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:43.025 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.025 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.996+0000 7fa9f1826640 1 librados: starting msgr at 2023-10-30T17:31:43.025 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.996+0000 7fa9f1826640 1 librados: starting objecter 2023-10-30T17:31:43.025 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 1 Processor -- start 2023-10-30T17:31:43.025 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 1 -- start start 2023-10-30T17:31:43.026 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 1 librados: setting wanted keys 2023-10-30T17:31:43.026 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 1 librados: calling monclient init 2023-10-30T17:31:43.026 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient: init 2023-10-30T17:31:43.026 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:43.026 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:31:43.027 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9ec06ec70 0x7fa9ec0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.027 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient(hunting): picked mon.noname-c con 0x7fa9ec06ec70 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:43.027 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 0x7fa9ec13c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.027 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient(hunting): picked mon.noname-b con 0x7fa9ec072060 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:43.028 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9ec13be60 0x7fa9ec129160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.028 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient(hunting): picked mon.noname-a con 0x7fa9ec13be60 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:43.028 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.028 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec13caf0 con 0x7fa9ec13be60 2023-10-30T17:31:43.028 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.029 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec12eea0 con 0x7fa9ec072060 2023-10-30T17:31:43.029 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.029 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec12e9c0 con 0x7fa9ec06ec70 2023-10-30T17:31:43.029 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:43.029 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.997+0000 7fa9f1826640 10 monclient(hunting): authenticate will time out at 1276.358841s 2023-10-30T17:31:43.030 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9e3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 0x7fa9ec13c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.030 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9eb7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9ec13be60 0x7fa9ec129160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.030 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9eaffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9ec06ec70 0x7fa9ec0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.030 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9e3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 0x7fa9ec13c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:43088/0 (socket says 172.21.15.177:43088) 2023-10-30T17:31:43.031 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9e3fff640 1 -- 172.21.15.177:0/4025759949 learned_addr learned my addr 172.21.15.177:0/4025759949 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:43.031 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9e3fff640 10 monclient(hunting): get_auth_request con 0x7fa9ec072060 auth_method 0 2023-10-30T17:31:43.031 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.031 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9e3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.031 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.032 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9eb7fe640 10 monclient(hunting): get_auth_request con 0x7fa9ec13be60 auth_method 0 2023-10-30T17:31:43.032 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.032 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9eb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.032 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9eb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.033 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9eaffd640 10 monclient(hunting): get_auth_request con 0x7fa9ec06ec70 auth_method 0 2023-10-30T17:31:43.033 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.033 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9eaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.033 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.998+0000 7fa9eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.033 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.999+0000 7fa9e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.034 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.999+0000 7fa9e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.034 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.999+0000 7fa9e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.034 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.999+0000 7fa9eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.034 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.999+0000 7fa9eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.035 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.999+0000 7fa9eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.035 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.999+0000 7fa9eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.035 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.999+0000 7fa9eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.035 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:42.999+0000 7fa9eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.035 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.001+0000 7fa9e3fff640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2023-10-30T17:31:43.036 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.001+0000 7fa9e3fff640 1 -- 172.21.15.177:0/4025759949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9ec06ec70 msgr2=0x7fa9ec0aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.036 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.001+0000 7fa9e3fff640 1 --2- 172.21.15.177:0/4025759949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9ec06ec70 0x7fa9ec0aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.036 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.001+0000 7fa9e3fff640 1 -- 172.21.15.177:0/4025759949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9ec13be60 msgr2=0x7fa9ec129160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.036 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.001+0000 7fa9e3fff640 1 --2- 172.21.15.177:0/4025759949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9ec13be60 0x7fa9ec129160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.036 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.001+0000 7fa9e3fff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:43.037 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.001+0000 7fa9e3fff640 1 monclient: found mon.noname-b 2023-10-30T17:31:43.037 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.001+0000 7fa9e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:43.037 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.001+0000 7fa9e3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:31:43.037 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.001+0000 7fa9e3fff640 1 -- 172.21.15.177:0/4025759949 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d4009400 con 0x7fa9ec072060 2023-10-30T17:31:43.038 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.004+0000 7fa9e3fff640 1 --2- 172.21.15.177:0/4025759949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 0x7fa9ec13c5b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4034040 tx=0x7fa9d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:43.038 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.004+0000 7fa9e8ff9640 10 client.?.objecter ms_handle_connect 0x7fa9ec072060 2023-10-30T17:31:43.038 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.004+0000 7fa9e8ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:43.038 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 1 -- 172.21.15.177:0/4025759949 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4032840 con 0x7fa9ec072060 2023-10-30T17:31:43.038 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.039 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:31:43.039 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 10 monclient: dump: 2023-10-30T17:31:43.039 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.039 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.040 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.040 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.040 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.040 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.040 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.041 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.041 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.041 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.041 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 10 monclient: _finish_auth 0 2023-10-30T17:31:43.041 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 10 monclient: _check_auth_tickets 2023-10-30T17:31:43.042 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:43.042 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 1 -- 172.21.15.177:0/4025759949 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d4031f00 con 0x7fa9ec072060 2023-10-30T17:31:43.042 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:43.042 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 1 -- 172.21.15.177:0/4025759949 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4032be0 con 0x7fa9ec072060 2023-10-30T17:31:43.042 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.043 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:31:43.043 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e8ff9640 10 monclient: dump: 2023-10-30T17:31:43.043 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.043 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.044 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.044 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.044 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.044 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.044 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.045 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.045 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.045 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.045 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9f1826640 5 monclient: authenticate success, global_id 4144 2023-10-30T17:31:43.045 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9f1826640 10 monclient: _renew_subs 2023-10-30T17:31:43.046 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9e9ffb640 4 set_mon_vals no callback set 2023-10-30T17:31:43.046 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9f1826640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:43.046 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9f1826640 1 -- 172.21.15.177:0/4025759949 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ec12ec40 con 0x7fa9ec072060 2023-10-30T17:31:43.046 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9f1826640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:43.047 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9f1826640 10 monclient: _renew_subs 2023-10-30T17:31:43.047 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9f1826640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:43.047 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9f1826640 1 -- 172.21.15.177:0/4025759949 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ec129880 con 0x7fa9ec072060 2023-10-30T17:31:43.047 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.007+0000 7fa9f1826640 1 librados: init done 2023-10-30T17:31:43.047 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.008+0000 7fa9f1826640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:43.048 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.008+0000 7fa9f1826640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:43.048 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.008+0000 7fa9f1826640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:43.048 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.008+0000 7fa9f1826640 1 -- 172.21.15.177:0/4025759949 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ec06af70 con 0x7fa9ec072060 2023-10-30T17:31:43.048 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.010+0000 7fa9e8ff9640 1 -- 172.21.15.177:0/4025759949 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa9d4005c90 con 0x7fa9ec072060 2023-10-30T17:31:43.048 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.010+0000 7fa9e8ff9640 1 -- 172.21.15.177:0/4025759949 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fa9d4006700 con 0x7fa9ec072060 2023-10-30T17:31:43.049 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.010+0000 7fa9e8ff9640 10 client.4144.objecter ms_dispatch 0x7fa9ec000b90 osd_map(7..7 src has 1..7) v4 2023-10-30T17:31:43.049 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.010+0000 7fa9e8ff9640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2023-10-30T17:31:43.049 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.010+0000 7fa9e8ff9640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2023-10-30T17:31:43.049 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.010+0000 7fa9e8ff9640 20 client.4144.objecter dump_active .. 0 homeless 2023-10-30T17:31:43.109 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.108+0000 7fa9e8ff9640 1 -- 172.21.15.177:0/4025759949 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa9ec06e610 con 0x7fa9ec072060 2023-10-30T17:31:43.110 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.108+0000 7fa9e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:43.110 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.108+0000 7fa9e8ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:43.180 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.179+0000 7fa9f1826640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2314fc35-d5d3-4609-853a-c0cbf5212ea7", "id": 5}] 2023-10-30T17:31:43.181 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.179+0000 7fa9f1826640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2314fc35-d5d3-4609-853a-c0cbf5212ea7", "id": 5}] 2023-10-30T17:31:43.181 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.179+0000 7fa9f1826640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:43.181 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.179+0000 7fa9f1826640 1 -- 172.21.15.177:0/4025759949 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2314fc35-d5d3-4609-853a-c0cbf5212ea7", "id": 5} v 0) v1 -- 0x7fa9ec0b1bf0 con 0x7fa9ec072060 2023-10-30T17:31:43.246 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.244+0000 7fa9e8ff9640 1 -- 172.21.15.177:0/4025759949 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2314fc35-d5d3-4609-853a-c0cbf5212ea7", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa9d4032d40 con 0x7fa9ec072060 2023-10-30T17:31:43.246 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.244+0000 7fa9e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2314fc35-d5d3-4609-853a-c0cbf5212ea7", "id": 5}] 2023-10-30T17:31:43.246 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.244+0000 7fa9e8ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:43.247 INFO:teuthology.orchestra.run.smithi177.stdout:5 2023-10-30T17:31:43.248 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 10 librados: watch_flush enter 2023-10-30T17:31:43.248 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 10 librados: watch_flush exit 2023-10-30T17:31:43.248 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 20 client.4144.objecter shutdown clearing up homeless session... 2023-10-30T17:31:43.249 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 10 client.4144.objecter successfully canceled tick 2023-10-30T17:31:43.249 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 10 monclient: shutdown 2023-10-30T17:31:43.249 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:43.249 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 1 -- 172.21.15.177:0/4025759949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 msgr2=0x7fa9ec13c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.249 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 1 --2- 172.21.15.177:0/4025759949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 0x7fa9ec13c5b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4034040 tx=0x7fa9d40046c0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.250 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 1 -- 172.21.15.177:0/4025759949 shutdown_connections 2023-10-30T17:31:43.250 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 1 --2- 172.21.15.177:0/4025759949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9ec072060 0x7fa9ec13c5b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.250 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 1 --2- 172.21.15.177:0/4025759949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9ec13be60 0x7fa9ec129160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.250 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.245+0000 7fa9f1826640 1 --2- 172.21.15.177:0/4025759949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9ec06ec70 0x7fa9ec0aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.250 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.246+0000 7fa9f1826640 1 -- 172.21.15.177:0/4025759949 >> 172.21.15.177:0/4025759949 conn(0x7fa9ec0b4020 msgr2=0x7fa9ec0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:43.251 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.246+0000 7fa9f1826640 1 -- 172.21.15.177:0/4025759949 shutdown_connections 2023-10-30T17:31:43.251 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.246+0000 7fa9f1826640 1 -- 172.21.15.177:0/4025759949 wait complete. 2023-10-30T17:31:43.251 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.246+0000 7fa9f1826640 1 librados: shutdown 2023-10-30T17:31:43.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 3e78cc0a-2199-401d-b956-382ab3af2ed1 6 2023-10-30T17:31:43.356 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.355+0000 7f44cb468640 10 monclient: get_monmap_and_config 2023-10-30T17:31:43.356 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.355+0000 7f44cb468640 10 monclient: build_initial_monmap 2023-10-30T17:31:43.357 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.355+0000 7f44cb468640 10 monclient: monmap: 2023-10-30T17:31:43.357 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:43.357 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.357 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:31:43.356244+0000 2023-10-30T17:31:43.357 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:31:43.356244+0000 2023-10-30T17:31:43.358 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:43.358 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.358 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:43.358 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:43.358 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:43.359 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:43.359 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:43.359 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.362 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.355+0000 7f44cb468640 1 Processor -- start 2023-10-30T17:31:43.362 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 1 -- start start 2023-10-30T17:31:43.362 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 10 monclient: init 2023-10-30T17:31:43.362 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:43.363 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-10-30T17:31:43.363 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44c4071900 addr v1:172.21.15.177:6789/0 2023-10-30T17:31:43.363 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44c406e4d0 addr v1:172.21.15.170:6789/0 2023-10-30T17:31:43.363 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406f3f0 0x7f44c413e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.363 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 10 monclient(hunting): picked mon.noname-b con 0x7f44c406f3f0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:43.364 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c4071470 con 0x7f44c406e4d0 2023-10-30T17:31:43.364 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c40ade40 con 0x7f44c4071900 2023-10-30T17:31:43.364 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.364 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c413c420 con 0x7f44c406f3f0 2023-10-30T17:31:43.364 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:43.365 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.356+0000 7f44cb468640 10 monclient(hunting): authenticate will time out at 1276.717702s 2023-10-30T17:31:43.365 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.357+0000 7f44c9a05640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406f3f0 0x7f44c413e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.365 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.357+0000 7f44c9204640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f44c4071900 0x7f44c4071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:38260/0 (socket says 172.21.15.177:38260) 2023-10-30T17:31:43.365 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.357+0000 7f44c9204640 1 -- 172.21.15.177:0/591710104 learned_addr learned my addr 172.21.15.177:0/591710104 (peer_addr_for_me v1:172.21.15.177:0/0) 2023-10-30T17:31:43.365 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.357+0000 7f44c9a05640 10 monclient(hunting): get_auth_request con 0x7f44c406f3f0 auth_method 0 2023-10-30T17:31:43.365 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.357+0000 7f44c9a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.366 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.357+0000 7f44c9a05640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.366 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.357+0000 7f44c9a05640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.366 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.358+0000 7f44c9a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.366 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.358+0000 7f44c9a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.366 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.358+0000 7f44c9a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.367 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.359+0000 7f44c9a05640 10 monclient(hunting): handle_auth_done global_id 4139 payload 274 2023-10-30T17:31:43.367 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.359+0000 7f44c9a05640 1 -- 172.21.15.177:0/591710104 >> v1:172.21.15.177:6789/0 conn(0x7f44c4071900 legacy=0x7f44c4071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.367 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.359+0000 7f44c9a05640 1 -- 172.21.15.177:0/591710104 >> v1:172.21.15.170:6789/0 conn(0x7f44c406e4d0 legacy=0x7f44c40ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.367 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.359+0000 7f44c9a05640 10 monclient: _finish_hunting 0 2023-10-30T17:31:43.367 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.359+0000 7f44c9a05640 1 monclient: found mon.noname-b 2023-10-30T17:31:43.368 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.359+0000 7f44c9a05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:43.368 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.359+0000 7f44c9a05640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:31:43.368 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.359+0000 7f44c9a05640 1 -- 172.21.15.177:0/591710104 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c406ebf0 con 0x7f44c406f3f0 2023-10-30T17:31:43.372 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.361+0000 7f44c9a05640 1 --2- 172.21.15.177:0/591710104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406f3f0 0x7f44c413e760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009dc0 tx=0x7f44bc02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffd10d8e6c61af3b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:43.372 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 1 -- 172.21.15.177:0/591710104 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc02cc40 con 0x7f44c406f3f0 2023-10-30T17:31:43.373 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.373 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:31:43.373 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 10 monclient: dump: 2023-10-30T17:31:43.373 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.373 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.373 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.374 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.374 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.374 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.374 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.374 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.375 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.375 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.375 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 10 monclient: _finish_auth 0 2023-10-30T17:31:43.375 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 10 monclient: _check_auth_tickets 2023-10-30T17:31:43.375 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:43.376 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 1 -- 172.21.15.177:0/591710104 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc02cda0 con 0x7f44c406f3f0 2023-10-30T17:31:43.376 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:43.376 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 1 -- 172.21.15.177:0/591710104 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc034660 con 0x7f44c406f3f0 2023-10-30T17:31:43.376 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.376 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:31:43.377 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44bbfff640 10 monclient: dump: 2023-10-30T17:31:43.377 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.377 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.377 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.377 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.377 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.378 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.378 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.378 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.378 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.378 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.379 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44cb468640 5 monclient: authenticate success, global_id 4139 2023-10-30T17:31:43.379 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44cb468640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:43.379 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.365+0000 7f44cb468640 4 set_mon_vals no callback set 2023-10-30T17:31:43.379 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 10 monclient: shutdown 2023-10-30T17:31:43.379 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:43.380 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 1 -- 172.21.15.177:0/591710104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406f3f0 msgr2=0x7f44c413e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.380 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 1 --2- 172.21.15.177:0/591710104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406f3f0 0x7f44c413e760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009dc0 tx=0x7f44bc02be90 comp rx=0 tx=0).stop 2023-10-30T17:31:43.380 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 1 -- 172.21.15.177:0/591710104 shutdown_connections 2023-10-30T17:31:43.380 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 1 --2- 172.21.15.177:0/591710104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406f3f0 0x7f44c413e760 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.380 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 1 -- 172.21.15.177:0/591710104 >> 172.21.15.177:0/591710104 conn(0x7f44c40b4020 msgr2=0x7f44c40b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:43.381 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 1 -- 172.21.15.177:0/591710104 shutdown_connections 2023-10-30T17:31:43.381 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 1 -- 172.21.15.177:0/591710104 wait complete. 2023-10-30T17:31:43.381 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 10 monclient: build_initial_monmap 2023-10-30T17:31:43.381 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.366+0000 7f44cb468640 10 monclient: monmap: 2023-10-30T17:31:43.381 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:43.382 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:43.382 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2023-10-30T17:31:43.382 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2023-10-30T17:31:43.382 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:43.382 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.382 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:43.383 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:43.383 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:43.383 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.383 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 librados: starting msgr at 2023-10-30T17:31:43.383 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 librados: starting objecter 2023-10-30T17:31:43.384 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 Processor -- start 2023-10-30T17:31:43.384 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 -- start start 2023-10-30T17:31:43.384 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 librados: setting wanted keys 2023-10-30T17:31:43.384 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 librados: calling monclient init 2023-10-30T17:31:43.384 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient: init 2023-10-30T17:31:43.384 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:43.385 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:31:43.385 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406e4d0 0x7f44c413e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.385 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient(hunting): picked mon.noname-c con 0x7f44c406e4d0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:43.385 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44c406f3f0 0x7f44c4138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.385 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient(hunting): picked mon.noname-b con 0x7f44c406f3f0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:43.386 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44c4071900 0x7f44c4141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.386 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient(hunting): picked mon.noname-a con 0x7f44c4071900 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:43.386 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.386 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c4138950 con 0x7f44c4071900 2023-10-30T17:31:43.386 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.387 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c4139d70 con 0x7f44c406f3f0 2023-10-30T17:31:43.387 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.387 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c41341a0 con 0x7f44c406e4d0 2023-10-30T17:31:43.387 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:43.387 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.367+0000 7f44cb468640 10 monclient(hunting): authenticate will time out at 1276.728848s 2023-10-30T17:31:43.387 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c8a03640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44c406f3f0 0x7f44c4138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.388 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c9204640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406e4d0 0x7f44c413e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.388 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c9a05640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44c4071900 0x7f44c4141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.388 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c8a03640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44c406f3f0 0x7f44c4138330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:43102/0 (socket says 172.21.15.177:43102) 2023-10-30T17:31:43.388 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c8a03640 1 -- 172.21.15.177:0/3151041573 learned_addr learned my addr 172.21.15.177:0/3151041573 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:43.388 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c8a03640 10 monclient(hunting): get_auth_request con 0x7f44c406f3f0 auth_method 0 2023-10-30T17:31:43.389 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c8a03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.389 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c8a03640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.389 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c8a03640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.389 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c9a05640 10 monclient(hunting): get_auth_request con 0x7f44c4071900 auth_method 0 2023-10-30T17:31:43.389 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c9a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.390 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c9a05640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.390 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c9a05640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.390 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c9204640 10 monclient(hunting): get_auth_request con 0x7f44c406e4d0 auth_method 0 2023-10-30T17:31:43.390 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c9204640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.390 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c9204640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.390 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.368+0000 7f44c9204640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.391 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.369+0000 7f44c8a03640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.391 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.369+0000 7f44c8a03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.391 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.369+0000 7f44c8a03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.391 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.369+0000 7f44c9a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.391 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.369+0000 7f44c9a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.392 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.369+0000 7f44c9a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.392 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.369+0000 7f44c9204640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.392 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.369+0000 7f44c9204640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.392 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.369+0000 7f44c9204640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.394 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.371+0000 7f44c9a05640 10 monclient(hunting): handle_auth_done global_id 4155 payload 824 2023-10-30T17:31:43.395 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.371+0000 7f44c9a05640 1 -- 172.21.15.177:0/3151041573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406e4d0 msgr2=0x7f44c413e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.395 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.371+0000 7f44c9a05640 1 --2- 172.21.15.177:0/3151041573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406e4d0 0x7f44c413e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.395 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.371+0000 7f44c9a05640 1 -- 172.21.15.177:0/3151041573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44c406f3f0 msgr2=0x7f44c4138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.395 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.371+0000 7f44c9a05640 1 --2- 172.21.15.177:0/3151041573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44c406f3f0 0x7f44c4138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.395 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.372+0000 7f44c9a05640 10 monclient: _finish_hunting 0 2023-10-30T17:31:43.396 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.372+0000 7f44c9a05640 1 monclient: found mon.noname-a 2023-10-30T17:31:43.396 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.372+0000 7f44c9a05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:43.396 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.372+0000 7f44c9a05640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:43.396 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.372+0000 7f44c9a05640 1 -- 172.21.15.177:0/3151041573 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc009a70 con 0x7f44c4071900 2023-10-30T17:31:43.396 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.374+0000 7f44c9a05640 1 --2- 172.21.15.177:0/3151041573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44c4071900 0x7f44c4141820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f44bc033000 tx=0x7f44bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:43.397 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.374+0000 7f44b9ffb640 10 client.?.objecter ms_handle_connect 0x7f44c4071900 2023-10-30T17:31:43.397 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.374+0000 7f44b9ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:43.397 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 1 -- 172.21.15.177:0/3151041573 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc040650 con 0x7f44c4071900 2023-10-30T17:31:43.397 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.397 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:43.398 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 10 monclient: dump: 2023-10-30T17:31:43.398 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.398 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.398 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.398 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.399 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.399 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.399 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.399 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.399 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.400 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.400 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 10 monclient: _finish_auth 0 2023-10-30T17:31:43.400 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:31:43.400 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:43.400 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 1 -- 172.21.15.177:0/3151041573 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc0407b0 con 0x7f44c4071900 2023-10-30T17:31:43.401 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44cb468640 5 monclient: authenticate success, global_id 4155 2023-10-30T17:31:43.401 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44cb468640 10 monclient: _renew_subs 2023-10-30T17:31:43.401 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44cb468640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:43.401 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44cb468640 1 -- 172.21.15.177:0/3151041573 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c41343c0 con 0x7f44c4071900 2023-10-30T17:31:43.401 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:43.402 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 1 -- 172.21.15.177:0/3151041573 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc040b40 con 0x7f44c4071900 2023-10-30T17:31:43.402 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.402 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:43.402 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44b9ffb640 10 monclient: dump: 2023-10-30T17:31:43.402 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.403 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.403 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.403 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.403 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.403 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.403 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.404 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.404 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.404 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.404 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.377+0000 7f44bb7fe640 4 set_mon_vals no callback set 2023-10-30T17:31:43.404 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.378+0000 7f44cb468640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:43.405 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.378+0000 7f44cb468640 10 monclient: _renew_subs 2023-10-30T17:31:43.405 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.378+0000 7f44cb468640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:43.405 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.378+0000 7f44cb468640 1 -- 172.21.15.177:0/3151041573 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c4134840 con 0x7f44c4071900 2023-10-30T17:31:43.405 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.378+0000 7f44cb468640 1 librados: init done 2023-10-30T17:31:43.405 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.378+0000 7f44cb468640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:43.406 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.379+0000 7f44cb468640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:43.406 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.379+0000 7f44cb468640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:43.406 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.379+0000 7f44cb468640 1 -- 172.21.15.177:0/3151041573 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c4067cd0 con 0x7f44c4071900 2023-10-30T17:31:43.476 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.475+0000 7f44b9ffb640 1 -- 172.21.15.177:0/3151041573 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f44bc052430 con 0x7f44c4071900 2023-10-30T17:31:43.480 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.479+0000 7f44b9ffb640 1 -- 172.21.15.177:0/3151041573 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f44bc03e9f0 con 0x7f44c4071900 2023-10-30T17:31:43.481 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.479+0000 7f44b9ffb640 10 client.4155.objecter ms_dispatch 0x7f44c4000b90 osd_map(8..8 src has 1..8) v4 2023-10-30T17:31:43.481 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.479+0000 7f44b9ffb640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2023-10-30T17:31:43.481 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.479+0000 7f44b9ffb640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2023-10-30T17:31:43.481 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.479+0000 7f44b9ffb640 20 client.4155.objecter dump_active .. 0 homeless 2023-10-30T17:31:43.482 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.479+0000 7f44b9ffb640 1 -- 172.21.15.177:0/3151041573 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f44bc044020 con 0x7f44c4071900 2023-10-30T17:31:43.482 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.479+0000 7f44b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:43.482 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.479+0000 7f44b9ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:43.553 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.551+0000 7f44cb468640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3e78cc0a-2199-401d-b956-382ab3af2ed1", "id": 6}] 2023-10-30T17:31:43.553 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.551+0000 7f44cb468640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3e78cc0a-2199-401d-b956-382ab3af2ed1", "id": 6}] 2023-10-30T17:31:43.554 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.551+0000 7f44cb468640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:43.554 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.551+0000 7f44cb468640 1 -- 172.21.15.177:0/3151041573 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3e78cc0a-2199-401d-b956-382ab3af2ed1", "id": 6} v 0) v1 -- 0x7f44c4070b60 con 0x7f44c4071900 2023-10-30T17:31:43.592 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.591+0000 7f44b9ffb640 1 -- 172.21.15.177:0/3151041573 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3e78cc0a-2199-401d-b956-382ab3af2ed1", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f44bc039070 con 0x7f44c4071900 2023-10-30T17:31:43.593 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.591+0000 7f44b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3e78cc0a-2199-401d-b956-382ab3af2ed1", "id": 6}] 2023-10-30T17:31:43.593 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.591+0000 7f44b9ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:43.593 INFO:teuthology.orchestra.run.smithi177.stdout:6 2023-10-30T17:31:43.595 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 10 librados: watch_flush enter 2023-10-30T17:31:43.595 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 10 librados: watch_flush exit 2023-10-30T17:31:43.595 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 20 client.4155.objecter shutdown clearing up homeless session... 2023-10-30T17:31:43.595 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 10 client.4155.objecter successfully canceled tick 2023-10-30T17:31:43.596 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 10 monclient: shutdown 2023-10-30T17:31:43.596 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:43.596 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 1 -- 172.21.15.177:0/3151041573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44c4071900 msgr2=0x7f44c4141820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.596 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 1 --2- 172.21.15.177:0/3151041573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44c4071900 0x7f44c4141820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f44bc033000 tx=0x7f44bc004a40 comp rx=0 tx=0).stop 2023-10-30T17:31:43.596 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 1 -- 172.21.15.177:0/3151041573 shutdown_connections 2023-10-30T17:31:43.597 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 1 --2- 172.21.15.177:0/3151041573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44c406f3f0 0x7f44c4138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.597 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 1 --2- 172.21.15.177:0/3151041573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44c4071900 0x7f44c4141820 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.597 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 1 --2- 172.21.15.177:0/3151041573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44c406e4d0 0x7f44c413e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.597 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 1 -- 172.21.15.177:0/3151041573 >> 172.21.15.177:0/3151041573 conn(0x7f44c40b4020 msgr2=0x7f44c413b800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:43.598 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 1 -- 172.21.15.177:0/3151041573 shutdown_connections 2023-10-30T17:31:43.598 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 1 -- 172.21.15.177:0/3151041573 wait complete. 2023-10-30T17:31:43.598 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.592+0000 7f44cb468640 1 librados: shutdown 2023-10-30T17:31:43.604 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 7388842d-4347-4fd7-b258-09c16109727d 7 2023-10-30T17:31:43.703 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.701+0000 7f7ef555a640 10 monclient: get_monmap_and_config 2023-10-30T17:31:43.703 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.701+0000 7f7ef555a640 10 monclient: build_initial_monmap 2023-10-30T17:31:43.703 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.701+0000 7f7ef555a640 10 monclient: monmap: 2023-10-30T17:31:43.704 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:43.704 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.704 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:31:43.702797+0000 2023-10-30T17:31:43.704 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:31:43.702797+0000 2023-10-30T17:31:43.704 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:43.704 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.705 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:43.705 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:43.705 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:43.705 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:43.705 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:43.706 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.708 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.702+0000 7f7ef555a640 1 Processor -- start 2023-10-30T17:31:43.709 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.702+0000 7f7ef555a640 1 -- start start 2023-10-30T17:31:43.709 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.702+0000 7f7ef555a640 10 monclient: init 2023-10-30T17:31:43.709 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:43.709 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-10-30T17:31:43.709 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ef0070050 addr v1:172.21.15.170:6789/0 2023-10-30T17:31:43.709 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0071140 0x7f7ef013c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.710 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 10 monclient(hunting): picked mon.noname-b con 0x7f7ef0071140 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:43.710 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7ef0070430 0x7f7ef013cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.710 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 10 monclient(hunting): picked mon.noname-a con 0x7f7ef0070430 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:43.710 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef0142cc0 con 0x7f7ef0070050 2023-10-30T17:31:43.710 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.711 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef0072310 con 0x7f7ef0070430 2023-10-30T17:31:43.711 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.711 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef00ac1a0 con 0x7f7ef0071140 2023-10-30T17:31:43.711 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:43.711 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ef555a640 10 monclient(hunting): authenticate will time out at 1277.064293s 2023-10-30T17:31:43.712 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7ee7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0071140 0x7f7ef013c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.712 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.703+0000 7f7eef7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7ef0070430 0x7f7ef013cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.712 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7ee7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0071140 0x7f7ef013c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.177:50894/0 (socket says 172.21.15.177:50894) 2023-10-30T17:31:43.712 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7ee7fff640 1 -- 172.21.15.177:0/4284198712 learned_addr learned my addr 172.21.15.177:0/4284198712 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:43.712 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7ee7fff640 10 monclient(hunting): get_auth_request con 0x7f7ef0071140 auth_method 0 2023-10-30T17:31:43.712 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7ee7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.713 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7ee7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.713 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7ee7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.713 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7eef7fe640 10 monclient(hunting): get_auth_request con 0x7f7ef0070430 auth_method 0 2023-10-30T17:31:43.713 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7eef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.713 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7eef7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.714 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7eef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.714 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7ee7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.714 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7ee7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.714 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.704+0000 7f7ee7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.714 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.705+0000 7f7eef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.715 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.705+0000 7f7eef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.715 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.705+0000 7f7eef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.715 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.706+0000 7f7ee7fff640 10 monclient(hunting): handle_auth_done global_id 4145 payload 274 2023-10-30T17:31:43.715 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.706+0000 7f7ee7fff640 1 -- 172.21.15.177:0/4284198712 >> v2:172.21.15.170:3300/0 conn(0x7f7ef0070430 msgr2=0x7f7ef013cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:43.715 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.706+0000 7f7ee7fff640 1 --2- 172.21.15.177:0/4284198712 >> v2:172.21.15.170:3300/0 conn(0x7f7ef0070430 0x7f7ef013cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.716 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.706+0000 7f7ee7fff640 1 -- 172.21.15.177:0/4284198712 >> v1:172.21.15.170:6789/0 conn(0x7f7ef0070050 legacy=0x7f7ef01431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.716 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.706+0000 7f7ee7fff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:43.716 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.706+0000 7f7ee7fff640 1 monclient: found mon.noname-b 2023-10-30T17:31:43.716 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.706+0000 7f7ee7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:43.716 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.706+0000 7f7ee7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:31:43.716 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.706+0000 7f7ee7fff640 1 -- 172.21.15.177:0/4284198712 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef013d320 con 0x7f7ef0071140 2023-10-30T17:31:43.721 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.707+0000 7f7ee7fff640 1 --2- 172.21.15.177:0/4284198712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0071140 0x7f7ef013c8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009870 tx=0x7f7ed802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d27083fe681820e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:43.721 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 1 -- 172.21.15.177:0/4284198712 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8031640 con 0x7f7ef0071140 2023-10-30T17:31:43.721 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.721 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:31:43.722 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 10 monclient: dump: 2023-10-30T17:31:43.722 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.722 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.722 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.722 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.723 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.723 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.723 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.723 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.723 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.724 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.724 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 10 monclient: _finish_auth 0 2023-10-30T17:31:43.724 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 10 monclient: _check_auth_tickets 2023-10-30T17:31:43.724 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:43.724 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 1 -- 172.21.15.177:0/4284198712 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed80317a0 con 0x7f7ef0071140 2023-10-30T17:31:43.725 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:43.725 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 1 -- 172.21.15.177:0/4284198712 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8031b30 con 0x7f7ef0071140 2023-10-30T17:31:43.725 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 5 monclient: authenticate success, global_id 4145 2023-10-30T17:31:43.725 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:43.725 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 4 set_mon_vals no callback set 2023-10-30T17:31:43.726 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.726 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:31:43.726 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7eee7fc640 10 monclient: dump: 2023-10-30T17:31:43.726 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.726 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.727 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.727 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.727 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.727 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.727 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.727 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.728 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.728 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.728 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 10 monclient: shutdown 2023-10-30T17:31:43.728 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:43.728 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 1 -- 172.21.15.177:0/4284198712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0071140 msgr2=0x7f7ef013c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.729 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 1 --2- 172.21.15.177:0/4284198712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0071140 0x7f7ef013c8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009870 tx=0x7f7ed802b8f0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.729 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 1 -- 172.21.15.177:0/4284198712 shutdown_connections 2023-10-30T17:31:43.729 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 1 --2- 172.21.15.177:0/4284198712 >> v2:172.21.15.170:3300/0 conn(0x7f7ef0070430 0x7f7ef013cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.729 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 1 --2- 172.21.15.177:0/4284198712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0071140 0x7f7ef013c8a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.729 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 1 -- 172.21.15.177:0/4284198712 >> 172.21.15.177:0/4284198712 conn(0x7f7ef00b52b0 msgr2=0x7f7ef00b5690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:43.730 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 1 -- 172.21.15.177:0/4284198712 shutdown_connections 2023-10-30T17:31:43.730 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.712+0000 7f7ef555a640 1 -- 172.21.15.177:0/4284198712 wait complete. 2023-10-30T17:31:43.730 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.713+0000 7f7ef555a640 10 monclient: build_initial_monmap 2023-10-30T17:31:43.730 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.713+0000 7f7ef555a640 10 monclient: monmap: 2023-10-30T17:31:43.730 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2023-10-30T17:31:43.731 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:43.731 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2023-10-30T17:31:43.731 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2023-10-30T17:31:43.731 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:43.731 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.731 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:43.732 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:43.732 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:43.732 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.732 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.713+0000 7f7ef555a640 1 librados: starting msgr at 2023-10-30T17:31:43.732 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.713+0000 7f7ef555a640 1 librados: starting objecter 2023-10-30T17:31:43.733 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.713+0000 7f7ef555a640 1 Processor -- start 2023-10-30T17:31:43.733 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 1 -- start start 2023-10-30T17:31:43.733 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 1 librados: setting wanted keys 2023-10-30T17:31:43.733 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 1 librados: calling monclient init 2023-10-30T17:31:43.733 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient: init 2023-10-30T17:31:43.734 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:43.734 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:31:43.734 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ef0070050 0x7f7ef00a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.734 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient(hunting): picked mon.noname-a con 0x7f7ef0070050 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:43.734 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0070430 0x7f7ef01d68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient(hunting): picked mon.noname-c con 0x7f7ef0070430 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:43.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ef0071140 0x7f7ef01d6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:43.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient(hunting): picked mon.noname-b con 0x7f7ef0071140 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:43.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.735 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef00aa460 con 0x7f7ef0070050 2023-10-30T17:31:43.736 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.736 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef01d63a0 con 0x7f7ef0071140 2023-10-30T17:31:43.736 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:43.736 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef00af040 con 0x7f7ef0070430 2023-10-30T17:31:43.736 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:43.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.714+0000 7f7ef555a640 10 monclient(hunting): authenticate will time out at 1277.075576s 2023-10-30T17:31:43.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7ee7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0070430 0x7f7ef01d68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eef7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ef0071140 0x7f7ef01d6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eeeffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ef0070050 0x7f7ef00a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:43.737 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eeeffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ef0070050 0x7f7ef00a9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.177:44400/0 (socket says 172.21.15.177:44400) 2023-10-30T17:31:43.738 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eeeffd640 1 -- 172.21.15.177:0/1850036556 learned_addr learned my addr 172.21.15.177:0/1850036556 (peer_addr_for_me v2:172.21.15.177:0/0) 2023-10-30T17:31:43.738 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eeeffd640 10 monclient(hunting): get_auth_request con 0x7f7ef0070050 auth_method 0 2023-10-30T17:31:43.738 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.738 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eeeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.738 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eeeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7ee7fff640 10 monclient(hunting): get_auth_request con 0x7f7ef0070430 auth_method 0 2023-10-30T17:31:43.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7ee7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7ee7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7ee7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eef7fe640 10 monclient(hunting): get_auth_request con 0x7f7ef0071140 auth_method 0 2023-10-30T17:31:43.739 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:43.740 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eef7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:43.740 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:43.740 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eeeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.740 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eeeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.740 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7eeeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.740 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7ee7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.741 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7ee7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.741 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.715+0000 7f7ee7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.741 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.716+0000 7f7eef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:43.741 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.716+0000 7f7eef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:43.741 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.716+0000 7f7eef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:43.742 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.718+0000 7f7eeeffd640 10 monclient(hunting): handle_auth_done global_id 4164 payload 824 2023-10-30T17:31:43.742 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.718+0000 7f7eeeffd640 1 -- 172.21.15.177:0/1850036556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0070430 msgr2=0x7f7ef01d68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.742 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.718+0000 7f7eeeffd640 1 --2- 172.21.15.177:0/1850036556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0070430 0x7f7ef01d68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.742 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.718+0000 7f7eeeffd640 1 -- 172.21.15.177:0/1850036556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ef0071140 msgr2=0x7f7ef01d6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.742 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.718+0000 7f7eeeffd640 1 --2- 172.21.15.177:0/1850036556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ef0071140 0x7f7ef01d6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.743 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.718+0000 7f7eeeffd640 10 monclient: _finish_hunting 0 2023-10-30T17:31:43.743 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.718+0000 7f7eeeffd640 1 monclient: found mon.noname-a 2023-10-30T17:31:43.743 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.718+0000 7f7eeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:43.743 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.718+0000 7f7eeeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:43.743 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.718+0000 7f7eeeffd640 1 -- 172.21.15.177:0/1850036556 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed80094d0 con 0x7f7ef0070050 2023-10-30T17:31:43.746 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.720+0000 7f7eeeffd640 1 --2- 172.21.15.177:0/1850036556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ef0070050 0x7f7ef00a9690 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7ee000c450 tx=0x7f7ee000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:43.746 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.720+0000 7f7ee77fe640 10 client.?.objecter ms_handle_connect 0x7f7ef0070050 2023-10-30T17:31:43.747 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.720+0000 7f7ee77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:43.747 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 1 -- 172.21.15.177:0/1850036556 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee00052d0 con 0x7f7ef0070050 2023-10-30T17:31:43.747 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.747 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:43.747 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 10 monclient: dump: 2023-10-30T17:31:43.748 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.748 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.748 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.748 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.748 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.748 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.749 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.749 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.749 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.749 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.749 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 10 monclient: _finish_auth 0 2023-10-30T17:31:43.750 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 10 monclient: _check_auth_tickets 2023-10-30T17:31:43.750 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:43.750 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 1 -- 172.21.15.177:0/1850036556 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee0005430 con 0x7f7ef0070050 2023-10-30T17:31:43.751 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:43.751 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 1 -- 172.21.15.177:0/1850036556 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0012870 con 0x7f7ef0070050 2023-10-30T17:31:43.751 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:43.751 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:43.751 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ee77fe640 10 monclient: dump: 2023-10-30T17:31:43.752 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2023-10-30T17:31:43.752 INFO:teuthology.orchestra.run.smithi177.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:43.752 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.752 INFO:teuthology.orchestra.run.smithi177.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:43.752 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:43.752 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2023-10-30T17:31:43.753 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:43.753 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:43.753 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:43.753 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T17:31:43.753 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ef555a640 5 monclient: authenticate success, global_id 4164 2023-10-30T17:31:43.754 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ef555a640 10 monclient: _renew_subs 2023-10-30T17:31:43.754 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ef555a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:43.754 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ef555a640 1 -- 172.21.15.177:0/1850036556 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef00ad6d0 con 0x7f7ef0070050 2023-10-30T17:31:43.754 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7eedffb640 4 set_mon_vals no callback set 2023-10-30T17:31:43.754 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ef555a640 10 client.4164.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:43.755 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ef555a640 10 monclient: _renew_subs 2023-10-30T17:31:43.755 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ef555a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:43.755 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ef555a640 1 -- 172.21.15.177:0/1850036556 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef00a9db0 con 0x7f7ef0070050 2023-10-30T17:31:43.755 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.724+0000 7f7ef555a640 1 librados: init done 2023-10-30T17:31:43.755 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.725+0000 7f7ef555a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:43.755 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.725+0000 7f7ef555a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:43.756 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.725+0000 7f7ef555a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:43.756 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.725+0000 7f7ef555a640 1 -- 172.21.15.177:0/1850036556 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef0067cd0 con 0x7f7ef0070050 2023-10-30T17:31:43.820 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.819+0000 7f7ee77fe640 1 -- 172.21.15.177:0/1850036556 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7ee0012cd0 con 0x7f7ef0070050 2023-10-30T17:31:43.827 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.825+0000 7f7ee77fe640 1 -- 172.21.15.177:0/1850036556 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7ee00035f0 con 0x7f7ef0070050 2023-10-30T17:31:43.827 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.825+0000 7f7ee77fe640 10 client.4164.objecter ms_dispatch 0x7f7ef0000b90 osd_map(9..9 src has 1..9) v4 2023-10-30T17:31:43.827 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.825+0000 7f7ee77fe640 3 client.4164.objecter handle_osd_map got epochs [9,9] > 0 2023-10-30T17:31:43.827 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.825+0000 7f7ee77fe640 3 client.4164.objecter handle_osd_map decoding full epoch 9 2023-10-30T17:31:43.828 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.825+0000 7f7ee77fe640 20 client.4164.objecter dump_active .. 0 homeless 2023-10-30T17:31:43.828 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.825+0000 7f7ee77fe640 1 -- 172.21.15.177:0/1850036556 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f7ee0007040 con 0x7f7ef0070050 2023-10-30T17:31:43.828 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.825+0000 7f7ee77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:43.828 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.825+0000 7f7ee77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:43.833 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:43.832+0000 7f2b9a22b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T17:31:43.901 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.899+0000 7f7ef555a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7388842d-4347-4fd7-b258-09c16109727d", "id": 7}] 2023-10-30T17:31:43.901 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.899+0000 7f7ef555a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7388842d-4347-4fd7-b258-09c16109727d", "id": 7}] 2023-10-30T17:31:43.901 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.899+0000 7f7ef555a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:43.901 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.899+0000 7f7ef555a640 1 -- 172.21.15.177:0/1850036556 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7388842d-4347-4fd7-b258-09c16109727d", "id": 7} v 0) v1 -- 0x7f7ef0066070 con 0x7f7ef0070050 2023-10-30T17:31:43.934 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.932+0000 7f7ee77fe640 1 -- 172.21.15.177:0/1850036556 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7388842d-4347-4fd7-b258-09c16109727d", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7ee002a0b0 con 0x7f7ef0070050 2023-10-30T17:31:43.934 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.932+0000 7f7ee77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7388842d-4347-4fd7-b258-09c16109727d", "id": 7}] 2023-10-30T17:31:43.934 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.932+0000 7f7ee77fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:43.935 INFO:teuthology.orchestra.run.smithi177.stdout:7 2023-10-30T17:31:43.936 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 10 librados: watch_flush enter 2023-10-30T17:31:43.936 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 10 librados: watch_flush exit 2023-10-30T17:31:43.936 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 20 client.4164.objecter shutdown clearing up homeless session... 2023-10-30T17:31:43.936 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 10 client.4164.objecter successfully canceled tick 2023-10-30T17:31:43.937 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 10 monclient: shutdown 2023-10-30T17:31:43.937 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:43.937 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 1 -- 172.21.15.177:0/1850036556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ef0070050 msgr2=0x7f7ef00a9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:43.937 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 1 --2- 172.21.15.177:0/1850036556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ef0070050 0x7f7ef00a9690 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7ee000c450 tx=0x7f7ee000c910 comp rx=0 tx=0).stop 2023-10-30T17:31:43.937 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 1 -- 172.21.15.177:0/1850036556 shutdown_connections 2023-10-30T17:31:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 1 --2- 172.21.15.177:0/1850036556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ef0071140 0x7f7ef01d6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 1 --2- 172.21.15.177:0/1850036556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ef0070430 0x7f7ef01d68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 1 --2- 172.21.15.177:0/1850036556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ef0070050 0x7f7ef00a9690 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 1 -- 172.21.15.177:0/1850036556 >> 172.21.15.177:0/1850036556 conn(0x7f7ef00b52b0 msgr2=0x7f7ef0138620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:43.938 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 1 -- 172.21.15.177:0/1850036556 shutdown_connections 2023-10-30T17:31:43.939 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 1 -- 172.21.15.177:0/1850036556 wait complete. 2023-10-30T17:31:43.939 INFO:teuthology.orchestra.run.smithi177.stderr:2023-10-30T17:31:43.933+0000 7f7ef555a640 1 librados: shutdown 2023-10-30T17:31:43.945 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T17:31:43.946 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T17:31:43.946 DEBUG:teuthology.orchestra.run.smithi170:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-10-30T17:31:43.949 INFO:tasks.ceph.osd.0:Started 2023-10-30T17:31:43.949 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T17:31:43.949 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T17:31:43.950 DEBUG:teuthology.orchestra.run.smithi170:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-10-30T17:31:43.952 INFO:tasks.ceph.osd.1:Started 2023-10-30T17:31:43.953 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T17:31:43.953 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T17:31:43.953 DEBUG:teuthology.orchestra.run.smithi170:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-10-30T17:31:43.956 INFO:tasks.ceph.osd.2:Started 2023-10-30T17:31:43.956 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T17:31:43.956 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T17:31:43.957 DEBUG:teuthology.orchestra.run.smithi170:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-10-30T17:31:43.960 INFO:tasks.ceph.osd.3:Started 2023-10-30T17:31:43.960 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T17:31:43.961 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T17:31:43.961 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-10-30T17:31:43.963 INFO:tasks.ceph.osd.4:Started 2023-10-30T17:31:43.964 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T17:31:43.964 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T17:31:43.964 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-10-30T17:31:43.966 INFO:tasks.ceph.osd.5:Started 2023-10-30T17:31:43.967 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T17:31:43.967 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-30T17:31:43.968 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-10-30T17:31:43.970 INFO:tasks.ceph.osd.6:Started 2023-10-30T17:31:43.970 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-10-30T17:31:43.970 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-30T17:31:43.971 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-10-30T17:31:43.973 INFO:tasks.ceph.osd.7:Started 2023-10-30T17:31:43.973 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T17:31:44.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.117+0000 7f734a701640 10 monclient: get_monmap_and_config 2023-10-30T17:31:44.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.117+0000 7f734a701640 10 monclient: build_initial_monmap 2023-10-30T17:31:44.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.117+0000 7f734a701640 10 monclient: monmap: 2023-10-30T17:31:44.119 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:31:44.120 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:44.120 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:31:44.118381+0000 2023-10-30T17:31:44.120 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:31:44.118381+0000 2023-10-30T17:31:44.120 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:44.120 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:44.121 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:44.121 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:44.121 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:44.121 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:44.121 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:44.121 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:44.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.118+0000 7f734a701640 1 Processor -- start 2023-10-30T17:31:44.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.118+0000 7f734a701640 1 -- start start 2023-10-30T17:31:44.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.118+0000 7f734a701640 10 monclient: init 2023-10-30T17:31:44.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.119+0000 7f734a701640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:44.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.119+0000 7f734a701640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T17:31:44.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.119+0000 7f734a701640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f73440b7c90 0x7f73440a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:44.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.119+0000 7f734a701640 10 monclient(hunting): picked mon.noname-a con 0x7f73440b7c90 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:44.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.119+0000 7f734a701640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f734413eb60 addr v1:172.21.15.170:6789/0 2023-10-30T17:31:44.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.119+0000 7f734a701640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7344144f50 addr v1:172.21.15.177:6789/0 2023-10-30T17:31:44.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f734a701640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73440aa740 con 0x7f734413eb60 2023-10-30T17:31:44.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f734a701640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73440aa9b0 con 0x7f7344144f50 2023-10-30T17:31:44.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f734a701640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:44.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f734a701640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73440a9500 con 0x7f73440b7c90 2023-10-30T17:31:44.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f734a701640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:44.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f734a701640 10 monclient(hunting): authenticate will time out at 1286.443313s 2023-10-30T17:31:44.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f7343fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f73440b7c90 0x7f73440a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:44.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f7348c9e640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f7344144f50 0x7f7344145400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:43666/0 (socket says 172.21.15.170:43666) 2023-10-30T17:31:44.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f7348c9e640 1 -- 172.21.15.170:0/2313116090 learned_addr learned my addr 172.21.15.170:0/2313116090 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:31:44.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f7343fff640 10 monclient(hunting): get_auth_request con 0x7f73440b7c90 auth_method 0 2023-10-30T17:31:44.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f7343fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:44.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f7343fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:44.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.120+0000 7f7343fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:44.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.121+0000 7f7343fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:44.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.121+0000 7f7343fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:44.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.121+0000 7f7343fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:44.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.124+0000 7f7343fff640 10 monclient(hunting): handle_auth_done global_id 4167 payload 274 2023-10-30T17:31:44.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.124+0000 7f7343fff640 1 -- 172.21.15.170:0/2313116090 >> v1:172.21.15.177:6789/0 conn(0x7f7344144f50 legacy=0x7f7344145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:44.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.124+0000 7f7343fff640 1 -- 172.21.15.170:0/2313116090 >> v1:172.21.15.170:6789/0 conn(0x7f734413eb60 legacy=0x7f734413efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:44.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.124+0000 7f7343fff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:44.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.124+0000 7f7343fff640 1 monclient: found mon.noname-a 2023-10-30T17:31:44.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.124+0000 7f7343fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:44.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.124+0000 7f7343fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:44.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.124+0000 7f7343fff640 1 -- 172.21.15.170:0/2313116090 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73440a9900 con 0x7f73440b7c90 2023-10-30T17:31:44.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.128+0000 7f7343fff640 1 --2- 172.21.15.170:0/2313116090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73440b7c90 0x7f73440a8b80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7334009720 tx=0x7f733402b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2740e48d8eb440 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:44.136 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:44.135+0000 7f2fb4c2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T17:31:44.137 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:44.136+0000 7f2b9a22b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T17:31:44.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 1 -- 172.21.15.170:0/2313116090 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7334030710 con 0x7f73440b7c90 2023-10-30T17:31:44.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:44.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:44.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 10 monclient: dump: 2023-10-30T17:31:44.143 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:31:44.143 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:44.143 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:44.143 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:44.144 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:44.144 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:44.144 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:44.144 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:44.144 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:44.145 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:44.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 10 monclient: _finish_auth 0 2023-10-30T17:31:44.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:31:44.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:44.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 1 -- 172.21.15.170:0/2313116090 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733402e050 con 0x7f73440b7c90 2023-10-30T17:31:44.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:44.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 1 -- 172.21.15.170:0/2313116090 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7334030a90 con 0x7f73440b7c90 2023-10-30T17:31:44.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:44.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:44.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f7342ffd640 10 monclient: dump: 2023-10-30T17:31:44.147 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:31:44.147 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:44.147 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:44.147 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:44.148 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:44.148 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:44.148 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:44.148 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:44.148 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:44.149 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:44.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f734a701640 5 monclient: authenticate success, global_id 4167 2023-10-30T17:31:44.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f734a701640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:44.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.137+0000 7f734a701640 4 set_mon_vals no callback set 2023-10-30T17:31:44.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.138+0000 7f734a701640 10 monclient: shutdown 2023-10-30T17:31:44.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.138+0000 7f734a701640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:44.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.138+0000 7f734a701640 1 -- 172.21.15.170:0/2313116090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73440b7c90 msgr2=0x7f73440a8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:44.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.138+0000 7f734a701640 1 --2- 172.21.15.170:0/2313116090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73440b7c90 0x7f73440a8b80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7334009720 tx=0x7f733402b690 comp rx=0 tx=0).stop 2023-10-30T17:31:44.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.138+0000 7f734a701640 1 -- 172.21.15.170:0/2313116090 shutdown_connections 2023-10-30T17:31:44.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.138+0000 7f734a701640 1 --2- 172.21.15.170:0/2313116090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73440b7c90 0x7f73440a8b80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:44.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.138+0000 7f734a701640 1 -- 172.21.15.170:0/2313116090 >> 172.21.15.170:0/2313116090 conn(0x7f7344071da0 msgr2=0x7f73440b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:44.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.139+0000 7f734a701640 1 -- 172.21.15.170:0/2313116090 shutdown_connections 2023-10-30T17:31:44.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.139+0000 7f734a701640 1 -- 172.21.15.170:0/2313116090 wait complete. 2023-10-30T17:31:44.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.140+0000 7f734a701640 10 monclient: build_initial_monmap 2023-10-30T17:31:44.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.140+0000 7f734a701640 10 monclient: monmap: 2023-10-30T17:31:44.151 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:31:44.152 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:44.152 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:31:44.152 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:31:44.152 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:44.152 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:44.153 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:44.153 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:44.153 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:44.153 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:44.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.140+0000 7f734a701640 1 librados: starting msgr at 2023-10-30T17:31:44.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.140+0000 7f734a701640 1 librados: starting objecter 2023-10-30T17:31:44.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.140+0000 7f734a701640 1 Processor -- start 2023-10-30T17:31:44.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.140+0000 7f734a701640 1 -- start start 2023-10-30T17:31:44.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.140+0000 7f734a701640 1 librados: setting wanted keys 2023-10-30T17:31:44.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.140+0000 7f734a701640 1 librados: calling monclient init 2023-10-30T17:31:44.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.140+0000 7f734a701640 10 monclient: init 2023-10-30T17:31:44.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.141+0000 7f734a701640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:44.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.141+0000 7f734a701640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:31:44.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.141+0000 7f734a701640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734413eb60 0x7f73441415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:44.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.142+0000 7f734a701640 10 monclient(hunting): picked mon.noname-a con 0x7f734413eb60 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:44.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.142+0000 7f734a701640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7344144f50 0x7f73441df0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:44.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.142+0000 7f734a701640 10 monclient(hunting): picked mon.noname-b con 0x7f7344144f50 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:44.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.142+0000 7f734a701640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7344141ae0 0x7f73441db500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:44.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f7343fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734413eb60 0x7f73441415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:44.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f734a701640 10 monclient(hunting): picked mon.noname-c con 0x7f7344141ae0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:44.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f734a701640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:44.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f734a701640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73441dec10 con 0x7f734413eb60 2023-10-30T17:31:44.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f734a701640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:44.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f7343fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734413eb60 0x7f73441415a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:49022/0 (socket says 172.21.15.170:49022) 2023-10-30T17:31:44.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f7343fff640 1 -- 172.21.15.170:0/759281350 learned_addr learned my addr 172.21.15.170:0/759281350 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:31:44.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f7348c9e640 1 --2- 172.21.15.170:0/759281350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7344141ae0 0x7f73441db500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:44.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f73437fe640 1 --2- 172.21.15.170:0/759281350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7344144f50 0x7f73441df0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:44.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734413fd30 con 0x7f7344144f50 2023-10-30T17:31:44.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f734a701640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:44.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734413a850 con 0x7f7344141ae0 2023-10-30T17:31:44.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f734a701640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:44.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.143+0000 7f734a701640 10 monclient(hunting): authenticate will time out at 1286.467058s 2023-10-30T17:31:44.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f73437fe640 10 monclient(hunting): get_auth_request con 0x7f7344144f50 auth_method 0 2023-10-30T17:31:44.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f73437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:44.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f73437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:44.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f73437fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:44.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7343fff640 10 monclient(hunting): get_auth_request con 0x7f734413eb60 auth_method 0 2023-10-30T17:31:44.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7343fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:44.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7343fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:44.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7343fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:44.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7348c9e640 10 monclient(hunting): get_auth_request con 0x7f7344141ae0 auth_method 0 2023-10-30T17:31:44.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7348c9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:44.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7348c9e640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:44.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7348c9e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:44.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7348c9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:44.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7348c9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:44.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7348c9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:44.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7343fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:44.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7343fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:44.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f7343fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:44.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f73437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:44.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f73437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:44.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.144+0000 7f73437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:44.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.149+0000 7f7343fff640 10 monclient(hunting): handle_auth_done global_id 4170 payload 824 2023-10-30T17:31:44.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.149+0000 7f7343fff640 1 -- 172.21.15.170:0/759281350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7344141ae0 msgr2=0x7f73441db500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:44.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.149+0000 7f7343fff640 1 --2- 172.21.15.170:0/759281350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7344141ae0 0x7f73441db500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:44.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.149+0000 7f7343fff640 1 -- 172.21.15.170:0/759281350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7344144f50 msgr2=0x7f73441df0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:44.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.149+0000 7f7343fff640 1 --2- 172.21.15.170:0/759281350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7344144f50 0x7f73441df0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:44.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.149+0000 7f7343fff640 10 monclient: _finish_hunting 0 2023-10-30T17:31:44.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.149+0000 7f7343fff640 1 monclient: found mon.noname-a 2023-10-30T17:31:44.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.149+0000 7f7343fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:44.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.149+0000 7f7343fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:44.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.149+0000 7f7343fff640 1 -- 172.21.15.170:0/759281350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7334009400 con 0x7f734413eb60 2023-10-30T17:31:44.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.154+0000 7f7343fff640 1 --2- 172.21.15.170:0/759281350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734413eb60 0x7f73441415a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7334009720 tx=0x7f73340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:44.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.154+0000 7f7340ff9640 10 client.?.objecter ms_handle_connect 0x7f734413eb60 2023-10-30T17:31:44.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.154+0000 7f7340ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:44.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.159+0000 7f7340ff9640 1 -- 172.21.15.170:0/759281350 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7334032990 con 0x7f734413eb60 2023-10-30T17:31:44.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.159+0000 7f7340ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:44.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.159+0000 7f7340ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:44.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f7340ff9640 10 monclient: dump: 2023-10-30T17:31:44.177 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:31:44.177 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:44.177 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:44.177 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:44.178 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:44.178 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:44.178 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:44.178 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:44.178 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:44.179 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:44.179 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f7340ff9640 10 monclient: _finish_auth 0 2023-10-30T17:31:44.179 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f7340ff9640 10 monclient: _check_auth_tickets 2023-10-30T17:31:44.179 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f7340ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:44.179 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f7340ff9640 1 -- 172.21.15.170:0/759281350 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7334032af0 con 0x7f734413eb60 2023-10-30T17:31:44.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f734a701640 5 monclient: authenticate success, global_id 4170 2023-10-30T17:31:44.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f734a701640 10 monclient: _renew_subs 2023-10-30T17:31:44.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f734a701640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:44.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734413aad0 con 0x7f734413eb60 2023-10-30T17:31:44.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f734a701640 10 client.4170.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:44.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f734a701640 10 monclient: _renew_subs 2023-10-30T17:31:44.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f734a701640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:44.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7344140a00 con 0x7f734413eb60 2023-10-30T17:31:44.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.160+0000 7f734a701640 1 librados: init done 2023-10-30T17:31:44.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.161+0000 7f7340ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:44.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.161+0000 7f7340ff9640 1 -- 172.21.15.170:0/759281350 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7334032e80 con 0x7f734413eb60 2023-10-30T17:31:44.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.161+0000 7f73427fc640 4 set_mon_vals no callback set 2023-10-30T17:31:44.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.161+0000 7f7340ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:44.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.161+0000 7f7340ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:44.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.161+0000 7f7340ff9640 10 monclient: dump: 2023-10-30T17:31:44.182 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:31:44.183 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:44.183 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:44.183 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:44.183 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:44.183 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:44.184 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:44.184 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:44.184 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:44.184 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:31:44.184 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.162+0000 7f734a701640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:44.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.162+0000 7f734a701640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:44.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.162+0000 7f734a701640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:44.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.162+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7324005040 con 0x7f734413eb60 2023-10-30T17:31:44.195 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:44.194+0000 7f2b9a22b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T17:31:44.306 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:44.305+0000 7f2b9a22b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T17:31:44.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.339+0000 7f7340ff9640 1 -- 172.21.15.170:0/759281350 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7334039040 con 0x7f734413eb60 2023-10-30T17:31:44.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.339+0000 7f7340ff9640 1 -- 172.21.15.170:0/759281350 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7334051c90 con 0x7f734413eb60 2023-10-30T17:31:44.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.339+0000 7f7340ff9640 10 client.4170.objecter ms_dispatch 0x7f7344000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T17:31:44.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.339+0000 7f7340ff9640 3 client.4170.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T17:31:44.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.339+0000 7f7340ff9640 3 client.4170.objecter handle_osd_map decoding full epoch 10 2023-10-30T17:31:44.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.339+0000 7f7340ff9640 20 client.4170.objecter dump_active .. 0 homeless 2023-10-30T17:31:44.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.339+0000 7f7340ff9640 1 -- 172.21.15.170:0/759281350 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f7334043020 con 0x7f734413eb60 2023-10-30T17:31:44.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.339+0000 7f7340ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:44.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.339+0000 7f7340ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:44.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.468+0000 7f734a701640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:31:44.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.468+0000 7f734a701640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:31:44.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.468+0000 7f734a701640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:44.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.468+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73240055c0 con 0x7f734413eb60 2023-10-30T17:31:44.493 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:44.492+0000 7f2fb4c2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T17:31:44.551 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:44.550+0000 7f2fb4c2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T17:31:44.626 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:31:44.626 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":10,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:31:43.906314+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:31:44.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.624+0000 7f7340ff9640 1 -- 172.21.15.170:0/759281350 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f7334056090 con 0x7f734413eb60 2023-10-30T17:31:44.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.624+0000 7f7340ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:31:44.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.624+0000 7f7340ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:44.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 10 librados: watch_flush enter 2023-10-30T17:31:44.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 10 librados: watch_flush exit 2023-10-30T17:31:44.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 20 client.4170.objecter shutdown clearing up homeless session... 2023-10-30T17:31:44.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 10 client.4170.objecter successfully canceled tick 2023-10-30T17:31:44.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 10 monclient: shutdown 2023-10-30T17:31:44.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:44.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734413eb60 msgr2=0x7f73441415a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:44.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 1 --2- 172.21.15.170:0/759281350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734413eb60 0x7f73441415a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7334009720 tx=0x7f73340046c0 comp rx=0 tx=0).stop 2023-10-30T17:31:44.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 shutdown_connections 2023-10-30T17:31:44.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 1 --2- 172.21.15.170:0/759281350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7344144f50 0x7f73441df0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:44.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 1 --2- 172.21.15.170:0/759281350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7344141ae0 0x7f73441db500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:44.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 1 --2- 172.21.15.170:0/759281350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734413eb60 0x7f73441415a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:44.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.625+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 >> 172.21.15.170:0/759281350 conn(0x7f7344071da0 msgr2=0x7f73440b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:44.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.626+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 shutdown_connections 2023-10-30T17:31:44.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.626+0000 7f734a701640 1 -- 172.21.15.170:0/759281350 wait complete. 2023-10-30T17:31:44.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:31:44.626+0000 7f734a701640 1 librados: shutdown 2023-10-30T17:31:44.643 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T17:31:44.644 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T17:31:44.690 INFO:tasks.ceph.mgr.y.smithi170.stderr:2023-10-30T17:31:44.689+0000 7f2fb4c2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T17:31:45.450 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T17:31:45.599 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.597+0000 7f6d5f597640 10 monclient: get_monmap_and_config 2023-10-30T17:31:45.599 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.597+0000 7f6d5f597640 10 monclient: build_initial_monmap 2023-10-30T17:31:45.600 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.597+0000 7f6d5f597640 10 monclient: monmap: 2023-10-30T17:31:45.600 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:31:45.600 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:45.601 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:31:45.598092+0000 2023-10-30T17:31:45.601 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:31:45.598092+0000 2023-10-30T17:31:45.601 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:45.602 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:45.602 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:45.602 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:45.603 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:45.603 INFO:teuthology.misc.health.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:45.603 INFO:teuthology.misc.health.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:45.603 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:45.604 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.598+0000 7f6d5f597640 1 Processor -- start 2023-10-30T17:31:45.608 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.598+0000 7f6d5f597640 1 -- start start 2023-10-30T17:31:45.608 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.598+0000 7f6d5f597640 10 monclient: init 2023-10-30T17:31:45.608 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:45.608 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-10-30T17:31:45.609 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d580b44b0 addr v1:172.21.15.177:6789/0 2023-10-30T17:31:45.609 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5813f350 0x7f6d580a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:45.609 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 10 monclient(hunting): picked mon.noname-b con 0x7f6d5813f350 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:45.610 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6d581419d0 0x7f6d58141e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:45.610 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 10 monclient(hunting): picked mon.noname-a con 0x7f6d581419d0 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:45.610 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d580aa4b0 con 0x7f6d580b44b0 2023-10-30T17:31:45.611 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:45.611 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d580b4890 con 0x7f6d581419d0 2023-10-30T17:31:45.611 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:45.612 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d580aa790 con 0x7f6d5813f350 2023-10-30T17:31:45.612 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:45.612 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.599+0000 7f6d5f597640 10 monclient(hunting): authenticate will time out at 1287.922670s 2023-10-30T17:31:45.613 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.600+0000 7f6d5d333640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f6d580b44b0 0x7f6d5813ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:43668/0 (socket says 172.21.15.170:43668) 2023-10-30T17:31:45.613 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.600+0000 7f6d5d333640 1 -- 172.21.15.170:0/3503512046 learned_addr learned my addr 172.21.15.170:0/3503512046 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:31:45.613 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.603+0000 7f6d5db34640 1 --2- 172.21.15.170:0/3503512046 >> v2:172.21.15.170:3300/0 conn(0x7f6d581419d0 0x7f6d58141e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:45.614 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.604+0000 7f6d4ffff640 1 -- 172.21.15.170:0/3503512046 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524650689 0 0) 0x7f6d580aa4b0 con 0x7f6d580b44b0 2023-10-30T17:31:45.614 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.604+0000 7f6d4ffff640 10 monclient(hunting): _negotiate 2023-10-30T17:31:45.614 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.604+0000 7f6d4ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:45.615 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.604+0000 7f6d4ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:45.615 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.604+0000 7f6d4ffff640 10 monclient(hunting): my global_id is 4171 2023-10-30T17:31:45.615 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.604+0000 7f6d4ffff640 1 -- 172.21.15.170:0/3503512046 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d44003520 con 0x7f6d580b44b0 2023-10-30T17:31:45.615 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.604+0000 7f6d5db34640 10 monclient(hunting): get_auth_request con 0x7f6d581419d0 auth_method 0 2023-10-30T17:31:45.615 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.604+0000 7f6d5db34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:45.616 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.604+0000 7f6d5db34640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:45.616 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.604+0000 7f6d5db34640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:45.616 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.606+0000 7f6d5db34640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:45.616 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.606+0000 7f6d5db34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:45.616 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.606+0000 7f6d5db34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:45.617 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.608+0000 7f6d5db34640 10 monclient(hunting): handle_auth_done global_id 4179 payload 274 2023-10-30T17:31:45.617 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.608+0000 7f6d5db34640 1 -- 172.21.15.170:0/3503512046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5813f350 msgr2=0x7f6d580a8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:45.618 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.608+0000 7f6d5db34640 1 --2- 172.21.15.170:0/3503512046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5813f350 0x7f6d580a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:45.618 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.608+0000 7f6d5db34640 1 -- 172.21.15.170:0/3503512046 >> v1:172.21.15.177:6789/0 conn(0x7f6d580b44b0 legacy=0x7f6d5813ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:45.618 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.608+0000 7f6d5db34640 10 monclient: _finish_hunting 0 2023-10-30T17:31:45.618 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.608+0000 7f6d5db34640 1 monclient: found mon.noname-a 2023-10-30T17:31:45.618 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.608+0000 7f6d5db34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:45.619 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.608+0000 7f6d5db34640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:45.619 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.608+0000 7f6d5db34640 1 -- 172.21.15.170:0/3503512046 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d580a9200 con 0x7f6d581419d0 2023-10-30T17:31:45.619 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.615+0000 7f6d5db34640 1 --2- 172.21.15.170:0/3503512046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d581419d0 0x7f6d58141e80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6d50009d90 tx=0x7f6d5002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c990f77bf28f5d17 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:45.622 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.620+0000 7f6d4ffff640 1 -- 172.21.15.170:0/3503512046 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5002ccf0 con 0x7f6d581419d0 2023-10-30T17:31:45.622 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.620+0000 7f6d4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:45.623 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.620+0000 7f6d4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:45.623 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.620+0000 7f6d4ffff640 10 monclient: dump: 2023-10-30T17:31:45.623 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:31:45.623 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:45.623 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:45.624 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:45.624 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:45.624 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:45.624 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:45.624 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:45.624 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:45.625 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:45.625 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.620+0000 7f6d4ffff640 10 monclient: _finish_auth 0 2023-10-30T17:31:45.625 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.620+0000 7f6d4ffff640 10 monclient: _check_auth_tickets 2023-10-30T17:31:45.625 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.620+0000 7f6d4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:45.625 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.620+0000 7f6d5f597640 5 monclient: authenticate success, global_id 4179 2023-10-30T17:31:45.626 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.621+0000 7f6d5f597640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:31:45.628 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.623+0000 7f6d4ffff640 1 -- 172.21.15.170:0/3503512046 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5002ce50 con 0x7f6d581419d0 2023-10-30T17:31:45.628 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.623+0000 7f6d4ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:45.629 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.623+0000 7f6d5f597640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:45.629 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.623+0000 7f6d5f597640 4 set_mon_vals no callback set 2023-10-30T17:31:45.629 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.623+0000 7f6d4ffff640 1 -- 172.21.15.170:0/3503512046 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5002ccf0 con 0x7f6d581419d0 2023-10-30T17:31:45.629 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.623+0000 7f6d4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:45.629 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.623+0000 7f6d4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:45.630 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.623+0000 7f6d4ffff640 10 monclient: dump: 2023-10-30T17:31:45.630 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:31:45.630 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:45.630 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:45.630 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:45.631 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:45.631 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:45.631 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:45.631 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:45.631 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:45.631 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:45.632 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.624+0000 7f6d5f597640 10 monclient: shutdown 2023-10-30T17:31:45.632 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.624+0000 7f6d5f597640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:45.632 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.624+0000 7f6d5f597640 1 -- 172.21.15.170:0/3503512046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d581419d0 msgr2=0x7f6d58141e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:45.632 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.624+0000 7f6d5f597640 1 --2- 172.21.15.170:0/3503512046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d581419d0 0x7f6d58141e80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6d50009d90 tx=0x7f6d5002bf60 comp rx=0 tx=0).stop 2023-10-30T17:31:45.632 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.624+0000 7f6d5f597640 1 -- 172.21.15.170:0/3503512046 shutdown_connections 2023-10-30T17:31:45.633 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.624+0000 7f6d5f597640 1 --2- 172.21.15.170:0/3503512046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d581419d0 0x7f6d58141e80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:45.633 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.624+0000 7f6d5f597640 1 --2- 172.21.15.170:0/3503512046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5813f350 0x7f6d580a8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:45.633 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.624+0000 7f6d5f597640 1 -- 172.21.15.170:0/3503512046 >> 172.21.15.170:0/3503512046 conn(0x7f6d58071b60 msgr2=0x7f6d58071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:45.633 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.625+0000 7f6d5f597640 1 -- 172.21.15.170:0/3503512046 shutdown_connections 2023-10-30T17:31:45.633 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.625+0000 7f6d5f597640 1 -- 172.21.15.170:0/3503512046 wait complete. 2023-10-30T17:31:45.634 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.626+0000 7f6d5f597640 10 monclient: build_initial_monmap 2023-10-30T17:31:45.634 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.626+0000 7f6d5f597640 10 monclient: monmap: 2023-10-30T17:31:45.634 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:31:45.634 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:45.635 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2023-10-30T17:31:45.635 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2023-10-30T17:31:45.635 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:45.635 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:45.635 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:45.635 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:45.636 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:45.636 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:45.636 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.627+0000 7f6d5f597640 1 librados: starting msgr at 2023-10-30T17:31:45.636 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.627+0000 7f6d5f597640 1 librados: starting objecter 2023-10-30T17:31:45.636 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.627+0000 7f6d5f597640 1 Processor -- start 2023-10-30T17:31:45.636 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.627+0000 7f6d5f597640 1 -- start start 2023-10-30T17:31:45.637 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.627+0000 7f6d5f597640 1 librados: setting wanted keys 2023-10-30T17:31:45.637 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.627+0000 7f6d5f597640 1 librados: calling monclient init 2023-10-30T17:31:45.637 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.627+0000 7f6d5f597640 10 monclient: init 2023-10-30T17:31:45.641 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:45.641 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:31:45.641 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580b44b0 0x7f6d58137510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:45.641 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 10 monclient(hunting): picked mon.noname-a con 0x7f6d580b44b0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:45.641 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5813f350 0x7f6d5813a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:45.642 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 10 monclient(hunting): picked mon.noname-c con 0x7f6d5813f350 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:45.642 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d581419d0 0x7f6d58139a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:45.642 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 10 monclient(hunting): picked mon.noname-b con 0x7f6d581419d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:45.642 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:45.642 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5813af10 con 0x7f6d580b44b0 2023-10-30T17:31:45.643 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:45.643 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5813acb0 con 0x7f6d581419d0 2023-10-30T17:31:45.643 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:45.643 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d581d64a0 con 0x7f6d5813f350 2023-10-30T17:31:45.644 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:45.644 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.628+0000 7f6d5f597640 10 monclient(hunting): authenticate will time out at 1287.951561s 2023-10-30T17:31:45.644 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.629+0000 7f6d5d333640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580b44b0 0x7f6d58137510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:45.645 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.629+0000 7f6d5d333640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580b44b0 0x7f6d58137510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:49040/0 (socket says 172.21.15.170:49040) 2023-10-30T17:31:45.645 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.629+0000 7f6d5d333640 1 -- 172.21.15.170:0/1762653357 learned_addr learned my addr 172.21.15.170:0/1762653357 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:31:45.645 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.629+0000 7f6d5d333640 10 monclient(hunting): get_auth_request con 0x7f6d580b44b0 auth_method 0 2023-10-30T17:31:45.646 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.629+0000 7f6d5d333640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:45.646 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.629+0000 7f6d5d333640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:45.646 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.629+0000 7f6d5d333640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:45.647 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.630+0000 7f6d5d333640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:45.647 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.630+0000 7f6d5d333640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:45.647 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.630+0000 7f6d5d333640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:45.648 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.630+0000 7f6d5db34640 1 --2- 172.21.15.170:0/1762653357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d581419d0 0x7f6d58139a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:45.648 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.634+0000 7f6d5d333640 10 monclient(hunting): handle_auth_done global_id 4185 payload 824 2023-10-30T17:31:45.648 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.634+0000 7f6d5d333640 1 -- 172.21.15.170:0/1762653357 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5813f350 msgr2=0x7f6d5813a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:45.649 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.634+0000 7f6d5d333640 1 --2- 172.21.15.170:0/1762653357 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5813f350 0x7f6d5813a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:45.649 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.634+0000 7f6d5d333640 1 -- 172.21.15.170:0/1762653357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d581419d0 msgr2=0x7f6d58139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:45.649 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.634+0000 7f6d5d333640 1 --2- 172.21.15.170:0/1762653357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d581419d0 0x7f6d58139a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:45.650 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.634+0000 7f6d5d333640 10 monclient: _finish_hunting 0 2023-10-30T17:31:45.650 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.634+0000 7f6d5d333640 1 monclient: found mon.noname-a 2023-10-30T17:31:45.651 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.634+0000 7f6d5d333640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:45.651 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.634+0000 7f6d5d333640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:45.651 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.634+0000 7f6d5d333640 1 -- 172.21.15.170:0/1762653357 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d50009a70 con 0x7f6d580b44b0 2023-10-30T17:31:45.651 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.639+0000 7f6d5d333640 1 --2- 172.21.15.170:0/1762653357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580b44b0 0x7f6d58137510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800c500 tx=0x7f6d4800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:45.652 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.639+0000 7f6d4dffb640 10 client.?.objecter ms_handle_connect 0x7f6d580b44b0 2023-10-30T17:31:45.652 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.639+0000 7f6d4dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:45.653 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d4dffb640 1 -- 172.21.15.170:0/1762653357 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48005ce0 con 0x7f6d580b44b0 2023-10-30T17:31:45.653 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:45.654 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:45.654 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d4dffb640 10 monclient: dump: 2023-10-30T17:31:45.654 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:31:45.654 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:45.655 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:45.655 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:45.655 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:45.656 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:45.656 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:45.656 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:45.656 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:45.656 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:45.657 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d4dffb640 10 monclient: _finish_auth 0 2023-10-30T17:31:45.657 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d4dffb640 10 monclient: _check_auth_tickets 2023-10-30T17:31:45.657 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:45.657 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d5f597640 5 monclient: authenticate success, global_id 4185 2023-10-30T17:31:45.657 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d5f597640 10 monclient: _renew_subs 2023-10-30T17:31:45.658 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d5f597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:45.658 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d5f597640 1 -- 172.21.15.170:0/1762653357 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d58139fa0 con 0x7f6d580b44b0 2023-10-30T17:31:45.658 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d5f597640 10 client.4185.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:45.658 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d5f597640 10 monclient: _renew_subs 2023-10-30T17:31:45.659 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d5f597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:45.659 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d5f597640 1 -- 172.21.15.170:0/1762653357 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d581d6ed0 con 0x7f6d580b44b0 2023-10-30T17:31:45.659 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.646+0000 7f6d5f597640 1 librados: init done 2023-10-30T17:31:45.659 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.647+0000 7f6d5f597640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:45.659 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.648+0000 7f6d5f597640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:45.659 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.648+0000 7f6d5f597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:45.660 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.648+0000 7f6d5f597640 1 -- 172.21.15.170:0/1762653357 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d58067cd0 con 0x7f6d580b44b0 2023-10-30T17:31:45.679 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.677+0000 7f6d4dffb640 1 -- 172.21.15.170:0/1762653357 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d48005e40 con 0x7f6d580b44b0 2023-10-30T17:31:45.679 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.678+0000 7f6d4dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:45.679 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.678+0000 7f6d4dffb640 1 -- 172.21.15.170:0/1762653357 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d480072f0 con 0x7f6d580b44b0 2023-10-30T17:31:45.680 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.678+0000 7f6d4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:45.680 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.678+0000 7f6d4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:45.680 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.678+0000 7f6d4dffb640 10 monclient: dump: 2023-10-30T17:31:45.680 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:31:45.680 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:45.681 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:45.681 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:45.681 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:45.681 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:45.681 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:45.682 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:45.682 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:45.682 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:45.682 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.678+0000 7f6d4f7fe640 4 set_mon_vals no callback set 2023-10-30T17:31:45.845 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.843+0000 7f6d4dffb640 1 -- 172.21.15.170:0/1762653357 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7f6d48007510 con 0x7f6d580b44b0 2023-10-30T17:31:45.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.846+0000 7f6d4dffb640 1 -- 172.21.15.170:0/1762653357 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6d4801aa30 con 0x7f6d580b44b0 2023-10-30T17:31:45.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.846+0000 7f6d4dffb640 10 client.4185.objecter ms_dispatch 0x7f6d58000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T17:31:45.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.846+0000 7f6d4dffb640 3 client.4185.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T17:31:45.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.846+0000 7f6d4dffb640 3 client.4185.objecter handle_osd_map decoding full epoch 10 2023-10-30T17:31:45.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:45.846+0000 7f6d4dffb640 20 client.4185.objecter dump_active .. 0 homeless 2023-10-30T17:31:46.058 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.056+0000 7f6d4dffb640 1 -- 172.21.15.170:0/1762653357 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f6d480232f0 con 0x7f6d580b44b0 2023-10-30T17:31:46.058 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.056+0000 7f6d4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:46.058 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.056+0000 7f6d4dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:46.181 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.179+0000 7f6d5f597640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:31:46.181 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.179+0000 7f6d5f597640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:31:46.181 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.179+0000 7f6d5f597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:31:46.182 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.179+0000 7f6d5f597640 1 -- 172.21.15.170:0/1762653357 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d58066140 con 0x7f6d580b44b0 2023-10-30T17:31:46.208 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.206+0000 7f6d4dffb640 1 -- 172.21.15.170:0/1762653357 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mgrmap(e 3) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f6d48012920 con 0x7f6d580b44b0 2023-10-30T17:31:46.319 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.318+0000 7f6d4dffb640 1 -- 172.21.15.170:0/1762653357 <== mon.0 v2:172.21.15.170:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f6d48035c90 con 0x7f6d580b44b0 2023-10-30T17:31:46.319 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.318+0000 7f6d4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:31:46.320 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.318+0000 7f6d4dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:46.320 INFO:teuthology.misc.health.smithi170.stdout: 2023-10-30T17:31:46.320 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:31:43.906314+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:31:46.322 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 10 librados: watch_flush enter 2023-10-30T17:31:46.322 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 10 librados: watch_flush exit 2023-10-30T17:31:46.323 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 20 client.4185.objecter shutdown clearing up homeless session... 2023-10-30T17:31:46.323 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 10 client.4185.objecter successfully canceled tick 2023-10-30T17:31:46.323 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 10 monclient: shutdown 2023-10-30T17:31:46.323 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:46.323 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 1 -- 172.21.15.170:0/1762653357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580b44b0 msgr2=0x7f6d58137510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:46.324 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 1 --2- 172.21.15.170:0/1762653357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580b44b0 0x7f6d58137510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800c500 tx=0x7f6d4800c9c0 comp rx=0 tx=0).stop 2023-10-30T17:31:46.324 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 1 -- 172.21.15.170:0/1762653357 shutdown_connections 2023-10-30T17:31:46.324 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 1 --2- 172.21.15.170:0/1762653357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d581419d0 0x7f6d58139a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:46.324 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 1 --2- 172.21.15.170:0/1762653357 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5813f350 0x7f6d5813a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:46.324 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 1 --2- 172.21.15.170:0/1762653357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580b44b0 0x7f6d58137510 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800c500 tx=0x7f6d4800c9c0 comp rx=0 tx=0).stop 2023-10-30T17:31:46.325 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.319+0000 7f6d5f597640 1 -- 172.21.15.170:0/1762653357 >> 172.21.15.170:0/1762653357 conn(0x7f6d58071b60 msgr2=0x7f6d5813bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:46.325 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.320+0000 7f6d5f597640 1 -- 172.21.15.170:0/1762653357 shutdown_connections 2023-10-30T17:31:46.325 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.320+0000 7f6d5f597640 1 -- 172.21.15.170:0/1762653357 wait complete. 2023-10-30T17:31:46.325 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:46.320+0000 7f6d5f597640 1 librados: shutdown 2023-10-30T17:31:46.335 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T17:31:47.870 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T17:31:47.868+0000 7f2b3aebc640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T17:31:52.144 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:08.117 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.146 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:08.098 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.207 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:08.180 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.212 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:08.164 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.240 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:08.186 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.304 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:08.250 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.429 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:08.391 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.491 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:08.454 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.502 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:08.461 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.564 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:08.522 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.658 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:08.627 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.718 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:08.687 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.754 INFO:tasks.ceph.osd.4.smithi177.stderr:2023-10-30T17:31:52.744+0000 5dc2340 -1 Falling back to public interface 2023-10-30T17:31:52.766 INFO:tasks.ceph.osd.5.smithi177.stderr:2023-10-30T17:31:52.757+0000 5dc2340 -1 Falling back to public interface 2023-10-30T17:31:52.861 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:08.814 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.894 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:31:52.887+0000 5dc2340 -1 Falling back to public interface 2023-10-30T17:31:52.920 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:08.873 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.931 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:08.909 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:52.991 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:08.968 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.063 INFO:tasks.ceph.osd.6.smithi177.stderr:2023-10-30T17:31:53.055+0000 5dc2340 -1 Falling back to public interface 2023-10-30T17:31:53.114 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-30T17:31:53.107+0000 5dc2340 -1 Falling back to public interface 2023-10-30T17:31:53.141 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T17:31:53.222 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:09.173 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.229 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.228+0000 7f3b15fa9640 10 monclient: get_monmap_and_config 2023-10-30T17:31:53.229 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.228+0000 7f3b15fa9640 10 monclient: build_initial_monmap 2023-10-30T17:31:53.229 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.228+0000 7f3b15fa9640 10 monclient: monmap: 2023-10-30T17:31:53.229 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:31:53.230 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:53.230 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:31:53.229015+0000 2023-10-30T17:31:53.230 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:31:53.229015+0000 2023-10-30T17:31:53.230 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:53.231 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:53.231 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:31:53.231 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:31:53.231 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:31:53.231 INFO:teuthology.misc.health.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:31:53.232 INFO:teuthology.misc.health.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:31:53.232 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:53.235 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.228+0000 7f3b15fa9640 1 Processor -- start 2023-10-30T17:31:53.235 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.228+0000 7f3b15fa9640 1 -- start start 2023-10-30T17:31:53.235 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.228+0000 7f3b15fa9640 10 monclient: init 2023-10-30T17:31:53.235 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:53.236 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-10-30T17:31:53.236 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3b100b4340 0x7f3b100a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:53.236 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 10 monclient(hunting): picked mon.noname-a con 0x7f3b100b4340 addr v2:172.21.15.170:3300/0 2023-10-30T17:31:53.236 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3b100a9460 0x7f3b10144f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:53.236 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 10 monclient(hunting): picked mon.noname-c con 0x7f3b100a9460 addr v2:172.21.15.177:3300/0 2023-10-30T17:31:53.237 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b101454d0 0x7f3b100b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:53.237 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 10 monclient(hunting): picked mon.noname-b con 0x7f3b101454d0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:53.237 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:53.237 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b100b4720 con 0x7f3b100b4340 2023-10-30T17:31:53.237 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:53.238 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b100a8d20 con 0x7f3b100a9460 2023-10-30T17:31:53.238 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:53.238 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b100a8fd0 con 0x7f3b101454d0 2023-10-30T17:31:53.238 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:53.239 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.229+0000 7f3b15fa9640 10 monclient(hunting): authenticate will time out at 1295.552634s 2023-10-30T17:31:53.239 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0f7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3b100b4340 0x7f3b100a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:53.239 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b101454d0 0x7f3b100b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:53.239 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0effd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3b100a9460 0x7f3b10144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:53.239 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0f7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3b100b4340 0x7f3b100a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40854/0 (socket says 172.21.15.170:40854) 2023-10-30T17:31:53.240 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0f7fe640 1 -- 172.21.15.170:0/594431736 learned_addr learned my addr 172.21.15.170:0/594431736 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:31:53.240 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0f7fe640 10 monclient(hunting): get_auth_request con 0x7f3b100b4340 auth_method 0 2023-10-30T17:31:53.240 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:53.240 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:53.240 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:53.241 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0effd640 10 monclient(hunting): get_auth_request con 0x7f3b100a9460 auth_method 0 2023-10-30T17:31:53.241 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:53.241 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:53.241 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:53.241 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0ffff640 10 monclient(hunting): get_auth_request con 0x7f3b101454d0 auth_method 0 2023-10-30T17:31:53.242 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:53.242 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:53.242 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.230+0000 7f3b0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:53.242 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.231+0000 7f3b0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:53.242 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.231+0000 7f3b0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:53.243 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.231+0000 7f3b0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:53.243 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.231+0000 7f3b0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:53.243 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.231+0000 7f3b0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:53.243 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.231+0000 7f3b0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:53.243 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.231+0000 7f3b0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:53.244 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.231+0000 7f3b0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:53.244 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.231+0000 7f3b0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:53.244 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.232+0000 7f3b0f7fe640 10 monclient(hunting): handle_auth_done global_id 4212 payload 274 2023-10-30T17:31:53.244 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.232+0000 7f3b0f7fe640 1 -- 172.21.15.170:0/594431736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b101454d0 msgr2=0x7f3b100b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:53.244 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.232+0000 7f3b0f7fe640 1 --2- 172.21.15.170:0/594431736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b101454d0 0x7f3b100b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.244 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.232+0000 7f3b0f7fe640 1 -- 172.21.15.170:0/594431736 >> v2:172.21.15.177:3300/0 conn(0x7f3b100a9460 msgr2=0x7f3b10144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:31:53.245 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.232+0000 7f3b0f7fe640 1 --2- 172.21.15.170:0/594431736 >> v2:172.21.15.177:3300/0 conn(0x7f3b100a9460 0x7f3b10144f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.245 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.232+0000 7f3b0f7fe640 10 monclient: _finish_hunting 0 2023-10-30T17:31:53.245 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.232+0000 7f3b0f7fe640 1 monclient: found mon.noname-a 2023-10-30T17:31:53.245 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.232+0000 7f3b0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:53.245 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.232+0000 7f3b0f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:31:53.246 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.232+0000 7f3b0f7fe640 1 -- 172.21.15.170:0/594431736 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b100aa400 con 0x7f3b100b4340 2023-10-30T17:31:53.250 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.234+0000 7f3b0f7fe640 1 --2- 172.21.15.170:0/594431736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b100b4340 0x7f3b100a87e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b100a8560 tx=0x7f3b0002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28553e9747ac096 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:53.250 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 1 -- 172.21.15.170:0/594431736 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b000317c0 con 0x7f3b100b4340 2023-10-30T17:31:53.251 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:53.251 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:31:53.251 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 10 monclient: dump: 2023-10-30T17:31:53.251 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:31:53.251 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:53.252 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:53.252 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:53.252 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:53.252 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:53.252 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:53.253 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:53.253 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:53.253 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:53.253 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 10 monclient: _finish_auth 0 2023-10-30T17:31:53.253 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 10 monclient: _check_auth_tickets 2023-10-30T17:31:53.254 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:53.254 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 1 -- 172.21.15.170:0/594431736 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b00031920 con 0x7f3b100b4340 2023-10-30T17:31:53.254 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b15fa9640 5 monclient: authenticate success, global_id 4212 2023-10-30T17:31:53.254 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b15fa9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:31:53.254 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:53.255 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b15fa9640 10 monclient: get_monmap_and_config success 2023-10-30T17:31:53.255 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b15fa9640 4 set_mon_vals no callback set 2023-10-30T17:31:53.255 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 1 -- 172.21.15.170:0/594431736 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00031cb0 con 0x7f3b100b4340 2023-10-30T17:31:53.255 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:53.255 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:31:53.256 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.239+0000 7f3b0e7fc640 10 monclient: dump: 2023-10-30T17:31:53.256 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:31:53.256 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:53.256 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:53.257 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:53.257 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:53.257 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:53.257 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:53.257 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:53.258 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:53.258 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:53.258 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 10 monclient: shutdown 2023-10-30T17:31:53.258 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:53.258 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 1 -- 172.21.15.170:0/594431736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b100b4340 msgr2=0x7f3b100a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:53.259 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 1 --2- 172.21.15.170:0/594431736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b100b4340 0x7f3b100a87e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b100a8560 tx=0x7f3b0002b860 comp rx=0 tx=0).stop 2023-10-30T17:31:53.259 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 1 -- 172.21.15.170:0/594431736 shutdown_connections 2023-10-30T17:31:53.259 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 1 --2- 172.21.15.170:0/594431736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b101454d0 0x7f3b100b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.259 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 1 --2- 172.21.15.170:0/594431736 >> v2:172.21.15.177:3300/0 conn(0x7f3b100a9460 0x7f3b10144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.259 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 1 --2- 172.21.15.170:0/594431736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b100b4340 0x7f3b100a87e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.260 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 1 -- 172.21.15.170:0/594431736 >> 172.21.15.170:0/594431736 conn(0x7f3b10071980 msgr2=0x7f3b10071d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:53.260 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 1 -- 172.21.15.170:0/594431736 shutdown_connections 2023-10-30T17:31:53.260 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.240+0000 7f3b15fa9640 1 -- 172.21.15.170:0/594431736 wait complete. 2023-10-30T17:31:53.260 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.241+0000 7f3b15fa9640 10 monclient: build_initial_monmap 2023-10-30T17:31:53.260 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.241+0000 7f3b15fa9640 10 monclient: monmap: 2023-10-30T17:31:53.261 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:31:53.261 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:31:53.261 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2023-10-30T17:31:53.261 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2023-10-30T17:31:53.261 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:31:53.262 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:53.262 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:31:53.262 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:31:53.262 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:31:53.263 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:53.263 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.241+0000 7f3b15fa9640 1 librados: starting msgr at 2023-10-30T17:31:53.263 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.241+0000 7f3b15fa9640 1 librados: starting objecter 2023-10-30T17:31:53.263 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.242+0000 7f3b15fa9640 1 Processor -- start 2023-10-30T17:31:53.263 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.242+0000 7f3b15fa9640 1 -- start start 2023-10-30T17:31:53.263 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.242+0000 7f3b15fa9640 1 librados: setting wanted keys 2023-10-30T17:31:53.264 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.242+0000 7f3b15fa9640 1 librados: calling monclient init 2023-10-30T17:31:53.264 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.242+0000 7f3b15fa9640 10 monclient: init 2023-10-30T17:31:53.264 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.242+0000 7f3b15fa9640 10 monclient: _reopen_session rank -1 2023-10-30T17:31:53.264 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:31:53.264 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b100a9460 0x7f3b100aadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:53.265 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 10 monclient(hunting): picked mon.noname-a con 0x7f3b100a9460 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:31:53.265 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b100b4340 0x7f3b100abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:53.265 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 10 monclient(hunting): picked mon.noname-b con 0x7f3b100b4340 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:31:53.265 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b101454d0 0x7f3b101d6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:53.266 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 10 monclient(hunting): picked mon.noname-c con 0x7f3b101454d0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:31:53.266 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:53.266 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b10142430 con 0x7f3b100a9460 2023-10-30T17:31:53.266 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:53.266 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b101d67c0 con 0x7f3b100b4340 2023-10-30T17:31:53.267 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:31:53.267 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b10140960 con 0x7f3b101454d0 2023-10-30T17:31:53.267 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 10 monclient(hunting): _renew_subs 2023-10-30T17:31:53.267 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b15fa9640 10 monclient(hunting): authenticate will time out at 1295.566159s 2023-10-30T17:31:53.267 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b0effd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b100b4340 0x7f3b100abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:53.268 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b0ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b101454d0 0x7f3b101d6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:53.268 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b0f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b100a9460 0x7f3b100aadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:53.268 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b0effd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b100b4340 0x7f3b100abb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:59916/0 (socket says 172.21.15.170:59916) 2023-10-30T17:31:53.268 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b0effd640 1 -- 172.21.15.170:0/3538253352 learned_addr learned my addr 172.21.15.170:0/3538253352 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:31:53.268 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b0effd640 10 monclient(hunting): get_auth_request con 0x7f3b100b4340 auth_method 0 2023-10-30T17:31:53.269 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:53.269 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:53.269 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.243+0000 7f3b0effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:53.269 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0ffff640 10 monclient(hunting): get_auth_request con 0x7f3b101454d0 auth_method 0 2023-10-30T17:31:53.269 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:53.270 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:53.270 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:53.270 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0f7fe640 10 monclient(hunting): get_auth_request con 0x7f3b100a9460 auth_method 0 2023-10-30T17:31:53.270 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:31:53.270 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:31:53.271 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:31:53.271 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:53.271 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:53.271 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:53.271 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:53.272 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:53.272 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:53.272 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:31:53.272 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.244+0000 7f3b0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:31:53.272 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.245+0000 7f3b0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:31:53.273 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.247+0000 7f3b0effd640 10 monclient(hunting): handle_auth_done global_id 4207 payload 824 2023-10-30T17:31:53.273 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.247+0000 7f3b0effd640 1 -- 172.21.15.170:0/3538253352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b101454d0 msgr2=0x7f3b101d6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:53.273 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.247+0000 7f3b0effd640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b101454d0 0x7f3b101d6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.273 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.247+0000 7f3b0effd640 1 -- 172.21.15.170:0/3538253352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b100a9460 msgr2=0x7f3b100aadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:53.273 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.247+0000 7f3b0effd640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b100a9460 0x7f3b100aadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.274 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.247+0000 7f3b0effd640 10 monclient: _finish_hunting 0 2023-10-30T17:31:53.274 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.247+0000 7f3b0effd640 1 monclient: found mon.noname-b 2023-10-30T17:31:53.274 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.247+0000 7f3b0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:31:53.274 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.247+0000 7f3b0effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:31:53.274 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.247+0000 7f3b0effd640 1 -- 172.21.15.170:0/3538253352 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b000094d0 con 0x7f3b100b4340 2023-10-30T17:31:53.276 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.250+0000 7f3b0effd640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b100b4340 0x7f3b100abb60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3af800e890 tx=0x7f3af800ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:53.276 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.250+0000 7f3b07fff640 10 client.?.objecter ms_handle_connect 0x7f3b100b4340 2023-10-30T17:31:53.276 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.250+0000 7f3b07fff640 10 client.?.objecter resend_mon_ops 2023-10-30T17:31:53.277 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 1 -- 172.21.15.170:0/3538253352 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af800c940 con 0x7f3b100b4340 2023-10-30T17:31:53.277 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:53.277 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:31:53.277 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 10 monclient: dump: 2023-10-30T17:31:53.277 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:31:53.278 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:53.278 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:53.278 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:53.278 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:53.278 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:53.279 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:53.279 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:53.279 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:53.279 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:53.279 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 10 monclient: _finish_auth 0 2023-10-30T17:31:53.280 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 10 monclient: _check_auth_tickets 2023-10-30T17:31:53.280 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:31:53.280 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 1 -- 172.21.15.170:0/3538253352 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af800caa0 con 0x7f3b100b4340 2023-10-30T17:31:53.280 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b15fa9640 5 monclient: authenticate success, global_id 4207 2023-10-30T17:31:53.280 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b15fa9640 10 monclient: _renew_subs 2023-10-30T17:31:53.280 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b15fa9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:53.281 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b15fa9640 1 -- 172.21.15.170:0/3538253352 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b100ac0a0 con 0x7f3b100b4340 2023-10-30T17:31:53.281 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:31:53.281 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 1 -- 172.21.15.170:0/3538253352 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af800ce30 con 0x7f3b100b4340 2023-10-30T17:31:53.281 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:31:53.281 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:31:53.282 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b07fff640 10 monclient: dump: 2023-10-30T17:31:53.282 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:31:53.282 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:31:53.282 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:53.282 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:31:53.283 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:31:53.283 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:31:53.283 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:31:53.283 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:31:53.283 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:31:53.284 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:31:53.284 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b0dffb640 4 set_mon_vals no callback set 2023-10-30T17:31:53.284 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b15fa9640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:31:53.284 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b15fa9640 10 monclient: _renew_subs 2023-10-30T17:31:53.284 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b15fa9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:53.284 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b15fa9640 1 -- 172.21.15.170:0/3538253352 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b101d71e0 con 0x7f3b100b4340 2023-10-30T17:31:53.285 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.254+0000 7f3b15fa9640 1 librados: init done 2023-10-30T17:31:53.285 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.255+0000 7f3b15fa9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:53.285 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.255+0000 7f3b15fa9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:53.285 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.255+0000 7f3b15fa9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:53.285 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.255+0000 7f3b15fa9640 1 -- 172.21.15.170:0/3538253352 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1004e3c0 con 0x7f3b100b4340 2023-10-30T17:31:53.286 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:09.255 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.287 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.280+0000 7f3b07fff640 1 -- 172.21.15.170:0/3538253352 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f3af8013470 con 0x7f3b100b4340 2023-10-30T17:31:53.287 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.281+0000 7f3b07fff640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3af006c7c0 0x7f3af006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:31:53.287 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.281+0000 7f3b0f7fe640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3af006c7c0 0x7f3af006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:31:53.288 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.281+0000 7f3b0f7fe640 10 monclient: get_auth_request con 0x7f3af006c7c0 auth_method 0 2023-10-30T17:31:53.288 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.281+0000 7f3b0f7fe640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3af006c7c0 0x7f3af006ec80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3b00036000 tx=0x7f3b0003d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:31:53.288 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.281+0000 7f3b07fff640 10 client.4207.objecter ms_handle_connect 0x7f3af006c7c0 2023-10-30T17:31:53.295 INFO:tasks.ceph.osd.0.smithi170.stderr:2023-10-30T17:31:53.288+0000 5dc2340 -1 Falling back to public interface 2023-10-30T17:31:53.402 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:09.347 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.473 INFO:tasks.ceph.osd.2.smithi170.stderr:2023-10-30T17:31:53.463+0000 5dc2340 -1 Falling back to public interface 2023-10-30T17:31:53.561 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.559+0000 7f3b07fff640 1 -- 172.21.15.170:0/3538253352 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f3af8058ce0 con 0x7f3b100b4340 2023-10-30T17:31:53.561 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.559+0000 7f3b07fff640 10 client.4207.objecter ms_dispatch 0x7f3b10000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T17:31:53.562 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.559+0000 7f3b07fff640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T17:31:53.562 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.559+0000 7f3b07fff640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2023-10-30T17:31:53.562 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.560+0000 7f3b07fff640 20 client.4207.objecter dump_active .. 0 homeless 2023-10-30T17:31:53.563 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.562+0000 7f3b07fff640 1 -- 172.21.15.170:0/3538253352 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3af8055020 con 0x7f3b100b4340 2023-10-30T17:31:53.563 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.562+0000 7f3b07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:31:53.563 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.562+0000 7f3b07fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:31:53.568 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:09.527 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.578 INFO:tasks.ceph.osd.1.smithi170.stderr:2023-10-30T17:31:53.566+0000 5dc2340 -1 Falling back to public interface 2023-10-30T17:31:53.596 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:09.548 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.603 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:09.566 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.655 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:09.628 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.753 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:09.699 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.771 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:09.740 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.789 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.788+0000 7f3b15fa9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:31:53.789 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.788+0000 7f3b15fa9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:31:53.790 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.788+0000 7f3b15fa9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:31:53.790 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.788+0000 7f3b15fa9640 1 -- 172.21.15.170:0/3538253352 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b100aa770 con 0x7f3b100b4340 2023-10-30T17:31:53.842 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.840+0000 7f3b07fff640 1 -- 172.21.15.170:0/3538253352 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f3af8021090 con 0x7f3b100b4340 2023-10-30T17:31:53.842 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.840+0000 7f3b07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:31:53.843 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.840+0000 7f3b07fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:31:53.843 INFO:teuthology.misc.health.smithi170.stdout: 2023-10-30T17:31:53.843 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:31:43.906314+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:31:53.845 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 10 librados: watch_flush enter 2023-10-30T17:31:53.845 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 10 librados: watch_flush exit 2023-10-30T17:31:53.845 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 20 client.4207.objecter shutdown clearing up homeless session... 2023-10-30T17:31:53.846 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 10 client.4207.objecter successfully canceled tick 2023-10-30T17:31:53.846 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 -- 172.21.15.170:0/3538253352 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3af006c7c0 msgr2=0x7f3af006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:53.846 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3af006c7c0 0x7f3af006ec80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3b00036000 tx=0x7f3b0003d040 comp rx=0 tx=0).stop 2023-10-30T17:31:53.846 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 10 monclient: shutdown 2023-10-30T17:31:53.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:31:53.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 -- 172.21.15.170:0/3538253352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b100b4340 msgr2=0x7f3b100abb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:31:53.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b100b4340 0x7f3b100abb60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3af800e890 tx=0x7f3af800ed50 comp rx=0 tx=0).stop 2023-10-30T17:31:53.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 -- 172.21.15.170:0/3538253352 shutdown_connections 2023-10-30T17:31:53.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3af006c7c0 0x7f3af006ec80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b100b4340 0x7f3b100abb60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b101454d0 0x7f3b101d6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 --2- 172.21.15.170:0/3538253352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b100a9460 0x7f3b100aadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:31:53.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 -- 172.21.15.170:0/3538253352 >> 172.21.15.170:0/3538253352 conn(0x7f3b10071980 msgr2=0x7f3b1013dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:31:53.849 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 -- 172.21.15.170:0/3538253352 shutdown_connections 2023-10-30T17:31:53.849 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 -- 172.21.15.170:0/3538253352 wait complete. 2023-10-30T17:31:53.849 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:31:53.843+0000 7f3b15fa9640 1 librados: shutdown 2023-10-30T17:31:53.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T17:31:53.929 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:09.881 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.957 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:09.920 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.969 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:09.942 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:53.984 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:09.943 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.007 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:09.960 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.091 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:10.068 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.119 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.064 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.126 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:10.095 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.208 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:10.160 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.268 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:10.240 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.318 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:10.281 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.352 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:10.311 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.359 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:10.312 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.405 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.351 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.452 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:10.429 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.484 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:10.453 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.495 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:10.447 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.559 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:10.532 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.609 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:10.572 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.645 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:10.604 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.695 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.641 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.724 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:10.677 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.786 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:10.738 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.787 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:10.756 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.810 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:10.788 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.846 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:10.819 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.898 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:10.861 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.932 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:10.891 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:54.982 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.928 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.012 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:10.966 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.073 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:11.025 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.079 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:11.049 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.108 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:11.085 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.133 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:11.105 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.187 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:11.149 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.229 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:11.188 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.274 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:11.220 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.276 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:11.230 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.338 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:11.290 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.343 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:11.312 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.376 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:11.353 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.402 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:11.375 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.450 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:11.413 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.494 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:11.453 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.538 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:11.484 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.541 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:11.494 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.602 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:11.553 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.614 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:11.584 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.655 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:11.632 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.671 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:11.643 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.725 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:11.688 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.758 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:11.717 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.804 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:11.749 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.805 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:11.759 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.880 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:11.849 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:55.924 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:11.901 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.001 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:11.963 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.023 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:11.982 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.072 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:12.024 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.073 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:12.027 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.126 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:12.078 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.146 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:12.119 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.151 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:12.121 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.193 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:12.170 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.200 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:12.173 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.274 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:12.219 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.328 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:12.274 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.351 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:12.305 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.461 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:12.439 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.496 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:12.455 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.513 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:12.475 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.549 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:12.508 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.567 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:12.530 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.621 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:12.590 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.676 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:12.645 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.838 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:12.792 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.893 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:12.846 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.945 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:12.922 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:31:56.999 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:12.977 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:00.662 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T17:32:00.751 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.750+0000 7f2356de9640 10 monclient: get_monmap_and_config 2023-10-30T17:32:00.753 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.750+0000 7f2356de9640 10 monclient: build_initial_monmap 2023-10-30T17:32:00.753 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.750+0000 7f2356de9640 10 monclient: monmap: 2023-10-30T17:32:00.753 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:32:00.753 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:00.754 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:32:00.751181+0000 2023-10-30T17:32:00.754 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:32:00.751181+0000 2023-10-30T17:32:00.754 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:00.754 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:00.754 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:00.755 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:00.755 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:00.755 INFO:teuthology.misc.health.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:00.755 INFO:teuthology.misc.health.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:00.755 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:00.757 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.750+0000 7f2356de9640 1 Processor -- start 2023-10-30T17:32:00.758 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.751+0000 7f2356de9640 1 -- start start 2023-10-30T17:32:00.758 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.751+0000 7f2356de9640 10 monclient: init 2023-10-30T17:32:00.758 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.751+0000 7f2356de9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:00.758 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.751+0000 7f2356de9640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-10-30T17:32:00.758 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.751+0000 7f2356de9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23500b4340 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:00.759 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.751+0000 7f2356de9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f23501456a0 0x7f23500a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:00.759 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.751+0000 7f2356de9640 10 monclient(hunting): picked mon.noname-a con 0x7f23501456a0 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:00.759 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.752+0000 7f2356de9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23500a9460 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:00.759 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.752+0000 7f2356de9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23500a8fd0 con 0x7f23500a9460 2023-10-30T17:32:00.759 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.752+0000 7f2356de9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23500aa400 con 0x7f23500b4340 2023-10-30T17:32:00.760 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.752+0000 7f2356de9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:00.760 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.752+0000 7f2356de9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23500a8d20 con 0x7f23501456a0 2023-10-30T17:32:00.760 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.752+0000 7f2356de9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:00.760 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.752+0000 7f2356de9640 10 monclient(hunting): authenticate will time out at 1303.075148s 2023-10-30T17:32:00.760 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.752+0000 7f234ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f23501456a0 0x7f23500a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:00.761 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.752+0000 7f2355386640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f23500a9460 0x7f235013e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:47298/0 (socket says 172.21.15.170:47298) 2023-10-30T17:32:00.761 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.752+0000 7f2355386640 1 -- 172.21.15.170:0/2900858189 learned_addr learned my addr 172.21.15.170:0/2900858189 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:00.761 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.753+0000 7f234ffff640 10 monclient(hunting): get_auth_request con 0x7f23501456a0 auth_method 0 2023-10-30T17:32:00.761 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.753+0000 7f234ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:00.761 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.753+0000 7f234ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:00.762 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.753+0000 7f234ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:00.762 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.754+0000 7f234ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:00.762 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.754+0000 7f234ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:00.762 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.754+0000 7f234ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:00.762 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.756+0000 7f234f7fe640 1 -- 172.21.15.170:0/2900858189 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338525283 0 0) 0x7f23500aa400 con 0x7f23500b4340 2023-10-30T17:32:00.762 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.756+0000 7f234f7fe640 10 monclient(hunting): _negotiate 2023-10-30T17:32:00.763 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.756+0000 7f234f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:00.763 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.756+0000 7f234f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:00.763 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.756+0000 7f234f7fe640 10 monclient(hunting): my global_id is 4210 2023-10-30T17:32:00.763 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.756+0000 7f234f7fe640 1 -- 172.21.15.170:0/2900858189 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2340003520 con 0x7f23500b4340 2023-10-30T17:32:00.764 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.758+0000 7f234ffff640 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2023-10-30T17:32:00.764 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.758+0000 7f234ffff640 1 -- 172.21.15.170:0/2900858189 >> v1:172.21.15.177:6789/0 conn(0x7f23500b4340 legacy=0x7f2350144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:00.764 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.758+0000 7f234ffff640 1 -- 172.21.15.170:0/2900858189 >> v1:172.21.15.170:6789/0 conn(0x7f23500a9460 legacy=0x7f235013e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:00.764 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.758+0000 7f234ffff640 10 monclient: _finish_hunting 0 2023-10-30T17:32:00.765 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.758+0000 7f234ffff640 1 monclient: found mon.noname-a 2023-10-30T17:32:00.765 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.758+0000 7f234ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:00.765 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.759+0000 7f234ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:00.765 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.759+0000 7f234ffff640 1 -- 172.21.15.170:0/2900858189 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23500aa6e0 con 0x7f23501456a0 2023-10-30T17:32:00.765 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.763+0000 7f234ffff640 1 --2- 172.21.15.170:0/2900858189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23501456a0 0x7f23500a87e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f233c0097a0 tx=0x7f233c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3346219d357a07ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:00.768 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 1 -- 172.21.15.170:0/2900858189 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c031830 con 0x7f23501456a0 2023-10-30T17:32:00.768 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:00.768 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:00.769 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 10 monclient: dump: 2023-10-30T17:32:00.769 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:00.769 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:00.769 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:00.769 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:00.770 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:00.770 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:00.770 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:00.770 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:00.770 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:00.771 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:00.771 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 10 monclient: _finish_auth 0 2023-10-30T17:32:00.771 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:32:00.771 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:00.771 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 1 -- 172.21.15.170:0/2900858189 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233c031990 con 0x7f23501456a0 2023-10-30T17:32:00.772 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:00.772 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 1 -- 172.21.15.170:0/2900858189 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c031d20 con 0x7f23501456a0 2023-10-30T17:32:00.772 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:00.772 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:00.772 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.766+0000 7f234f7fe640 10 monclient: dump: 2023-10-30T17:32:00.772 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:00.773 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:00.773 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:00.773 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:00.773 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:00.773 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:00.774 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:00.774 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:00.774 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:00.774 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:00.774 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.767+0000 7f2356de9640 5 monclient: authenticate success, global_id 4218 2023-10-30T17:32:00.774 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.767+0000 7f2356de9640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:00.775 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.767+0000 7f2356de9640 4 set_mon_vals no callback set 2023-10-30T17:32:00.780 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.767+0000 7f2356de9640 10 monclient: shutdown 2023-10-30T17:32:00.781 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.767+0000 7f2356de9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:00.781 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.767+0000 7f2356de9640 1 -- 172.21.15.170:0/2900858189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23501456a0 msgr2=0x7f23500a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:00.781 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.767+0000 7f2356de9640 1 --2- 172.21.15.170:0/2900858189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23501456a0 0x7f23500a87e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f233c0097a0 tx=0x7f233c02b8d0 comp rx=0 tx=0).stop 2023-10-30T17:32:00.781 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.768+0000 7f2356de9640 1 -- 172.21.15.170:0/2900858189 shutdown_connections 2023-10-30T17:32:00.781 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.768+0000 7f2356de9640 1 --2- 172.21.15.170:0/2900858189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23501456a0 0x7f23500a87e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:00.782 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.768+0000 7f2356de9640 1 -- 172.21.15.170:0/2900858189 >> 172.21.15.170:0/2900858189 conn(0x7f2350071980 msgr2=0x7f2350071d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:00.782 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.768+0000 7f2356de9640 1 -- 172.21.15.170:0/2900858189 shutdown_connections 2023-10-30T17:32:00.782 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.768+0000 7f2356de9640 1 -- 172.21.15.170:0/2900858189 wait complete. 2023-10-30T17:32:00.782 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.769+0000 7f2356de9640 10 monclient: build_initial_monmap 2023-10-30T17:32:00.782 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.769+0000 7f2356de9640 10 monclient: monmap: 2023-10-30T17:32:00.783 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:32:00.783 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:00.783 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:00.783 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2023-10-30T17:32:00.783 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:00.784 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:00.784 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:00.784 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:00.784 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:00.784 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:00.785 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 1 librados: starting msgr at 2023-10-30T17:32:00.785 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 1 librados: starting objecter 2023-10-30T17:32:00.785 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 1 Processor -- start 2023-10-30T17:32:00.785 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 1 -- start start 2023-10-30T17:32:00.785 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 1 librados: setting wanted keys 2023-10-30T17:32:00.785 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 1 librados: calling monclient init 2023-10-30T17:32:00.786 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 10 monclient: init 2023-10-30T17:32:00.786 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:00.786 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:32:00.786 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23500a9460 0x7f23501425a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:00.786 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 10 monclient(hunting): picked mon.noname-a con 0x7f23500a9460 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:00.787 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.770+0000 7f2356de9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23500b4340 0x7f23501448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:00.787 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 10 monclient(hunting): picked mon.noname-c con 0x7f23500b4340 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:00.787 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23501456a0 0x7f235013f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:00.787 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 10 monclient(hunting): picked mon.noname-b con 0x7f23501456a0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:00.787 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:00.787 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2350142110 con 0x7f23500a9460 2023-10-30T17:32:00.788 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:00.788 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23500abc00 con 0x7f23501456a0 2023-10-30T17:32:00.788 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:00.788 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235013fd50 con 0x7f23500b4340 2023-10-30T17:32:00.788 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:00.789 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2356de9640 10 monclient(hunting): authenticate will time out at 1303.094291s 2023-10-30T17:32:00.789 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2354b85640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23500a9460 0x7f23501425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:00.789 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f234ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23500b4340 0x7f23501448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:00.789 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.771+0000 7f2355386640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23501456a0 0x7f235013f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:00.789 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2354b85640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23500a9460 0x7f23501425a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:44804/0 (socket says 172.21.15.170:44804) 2023-10-30T17:32:00.789 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2354b85640 1 -- 172.21.15.170:0/2116114004 learned_addr learned my addr 172.21.15.170:0/2116114004 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:00.790 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2354b85640 10 monclient(hunting): get_auth_request con 0x7f23500a9460 auth_method 0 2023-10-30T17:32:00.790 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2354b85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:00.790 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2354b85640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:00.790 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2354b85640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:00.790 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f234ffff640 10 monclient(hunting): get_auth_request con 0x7f23500b4340 auth_method 0 2023-10-30T17:32:00.791 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f234ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:00.791 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f234ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:00.791 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f234ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:00.791 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2355386640 10 monclient(hunting): get_auth_request con 0x7f23501456a0 auth_method 0 2023-10-30T17:32:00.791 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2355386640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:00.791 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2355386640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:00.792 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2355386640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:00.792 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2354b85640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:00.792 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2354b85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:00.792 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f2354b85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:00.792 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f234ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:00.793 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f234ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:00.793 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.772+0000 7f234ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:00.793 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.773+0000 7f2355386640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:00.793 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.773+0000 7f2355386640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:00.793 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.773+0000 7f2355386640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:00.793 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.775+0000 7f2354b85640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2023-10-30T17:32:00.794 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.775+0000 7f2354b85640 1 -- 172.21.15.170:0/2116114004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23500b4340 msgr2=0x7f23501448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:00.794 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.775+0000 7f2354b85640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23500b4340 0x7f23501448e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:00.794 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.775+0000 7f2354b85640 1 -- 172.21.15.170:0/2116114004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23501456a0 msgr2=0x7f235013f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:00.794 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.775+0000 7f2354b85640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23501456a0 0x7f235013f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:00.794 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.776+0000 7f2354b85640 10 monclient: _finish_hunting 0 2023-10-30T17:32:00.795 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.776+0000 7f2354b85640 1 monclient: found mon.noname-a 2023-10-30T17:32:00.795 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.776+0000 7f2354b85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:00.795 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.776+0000 7f2354b85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:00.795 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.776+0000 7f2354b85640 1 -- 172.21.15.170:0/2116114004 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233c009400 con 0x7f23500a9460 2023-10-30T17:32:00.795 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.777+0000 7f234d7fa640 10 client.?.objecter ms_handle_connect 0x7f23500a9460 2023-10-30T17:32:00.796 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.777+0000 7f234d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:00.796 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.777+0000 7f2354b85640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23500a9460 0x7f23501425a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f234400c490 tx=0x7f234400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:00.798 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 1 -- 172.21.15.170:0/2116114004 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440052d0 con 0x7f23500a9460 2023-10-30T17:32:00.799 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:00.799 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:00.799 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 10 monclient: dump: 2023-10-30T17:32:00.799 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:00.799 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:00.800 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:00.800 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:00.800 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:00.800 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:00.800 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:00.801 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:00.801 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:00.801 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:00.801 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 10 monclient: _finish_auth 0 2023-10-30T17:32:00.801 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 10 monclient: _check_auth_tickets 2023-10-30T17:32:00.802 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:00.802 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f2356de9640 5 monclient: authenticate success, global_id 4224 2023-10-30T17:32:00.802 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f2356de9640 10 monclient: _renew_subs 2023-10-30T17:32:00.802 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f2356de9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:00.802 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f2356de9640 1 -- 172.21.15.170:0/2116114004 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235013ffd0 con 0x7f23500a9460 2023-10-30T17:32:00.802 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 1 -- 172.21.15.170:0/2116114004 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2344005430 con 0x7f23500a9460 2023-10-30T17:32:00.803 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:00.803 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 1 -- 172.21.15.170:0/2116114004 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440052d0 con 0x7f23500a9460 2023-10-30T17:32:00.803 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:00.803 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:00.803 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234d7fa640 10 monclient: dump: 2023-10-30T17:32:00.804 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:00.804 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:00.804 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:00.804 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:00.804 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:00.805 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:00.805 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:00.805 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:00.805 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:00.805 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:00.806 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.780+0000 7f234effd640 4 set_mon_vals no callback set 2023-10-30T17:32:00.806 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.781+0000 7f2356de9640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:00.806 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.781+0000 7f2356de9640 10 monclient: _renew_subs 2023-10-30T17:32:00.806 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.781+0000 7f2356de9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:00.806 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.781+0000 7f2356de9640 1 -- 172.21.15.170:0/2116114004 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2350141700 con 0x7f23500a9460 2023-10-30T17:32:00.806 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.781+0000 7f2356de9640 1 librados: init done 2023-10-30T17:32:00.807 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.782+0000 7f2356de9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:00.807 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.782+0000 7f2356de9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:00.807 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.782+0000 7f2356de9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:00.807 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:00.782+0000 7f2356de9640 1 -- 172.21.15.170:0/2116114004 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f235004e3c0 con 0x7f23500a9460 2023-10-30T17:32:01.012 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.011+0000 7f234d7fa640 1 -- 172.21.15.170:0/2116114004 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f2344012630 con 0x7f23500a9460 2023-10-30T17:32:01.013 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.011+0000 7f234d7fa640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f233406c7d0 0x7f233406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:01.013 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.012+0000 7f234ffff640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f233406c7d0 0x7f233406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:01.013 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.012+0000 7f234ffff640 10 monclient: get_auth_request con 0x7f233406c7d0 auth_method 0 2023-10-30T17:32:01.014 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.012+0000 7f234d7fa640 10 client.4224.objecter ms_handle_connect 0x7f233406c7d0 2023-10-30T17:32:01.014 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.012+0000 7f234ffff640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f233406c7d0 0x7f233406ec90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f233c036000 tx=0x7f233c03c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:01.023 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.021+0000 7f234d7fa640 1 -- 172.21.15.170:0/2116114004 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f23440585d0 con 0x7f23500a9460 2023-10-30T17:32:01.023 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.021+0000 7f234d7fa640 10 client.4224.objecter ms_dispatch 0x7f2350000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T17:32:01.023 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.021+0000 7f234d7fa640 3 client.4224.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T17:32:01.023 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.021+0000 7f234d7fa640 3 client.4224.objecter handle_osd_map decoding full epoch 10 2023-10-30T17:32:01.023 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.021+0000 7f234d7fa640 20 client.4224.objecter dump_active .. 0 homeless 2023-10-30T17:32:01.095 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.094+0000 7f234d7fa640 1 -- 172.21.15.170:0/2116114004 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2344054910 con 0x7f23500a9460 2023-10-30T17:32:01.096 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.094+0000 7f234d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:01.096 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.094+0000 7f234d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:01.414 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.413+0000 7f2356de9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:01.414 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.413+0000 7f2356de9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:01.415 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.413+0000 7f2356de9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:01.415 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.413+0000 7f2356de9640 1 -- 172.21.15.170:0/2116114004 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2350145460 con 0x7f23500a9460 2023-10-30T17:32:01.429 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.428+0000 7f234d7fa640 1 -- 172.21.15.170:0/2116114004 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2344057f30 con 0x7f23500a9460 2023-10-30T17:32:01.429 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.428+0000 7f234d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:01.429 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.428+0000 7f234d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:01.430 INFO:teuthology.misc.health.smithi170.stdout: 2023-10-30T17:32:01.430 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:31:43.906314+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:32:01.432 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.430+0000 7f2356de9640 10 librados: watch_flush enter 2023-10-30T17:32:01.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.430+0000 7f2356de9640 10 librados: watch_flush exit 2023-10-30T17:32:01.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.430+0000 7f2356de9640 20 client.4224.objecter shutdown clearing up homeless session... 2023-10-30T17:32:01.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.430+0000 7f2356de9640 10 client.4224.objecter successfully canceled tick 2023-10-30T17:32:01.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.430+0000 7f2356de9640 1 -- 172.21.15.170:0/2116114004 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f233406c7d0 msgr2=0x7f233406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:01.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.430+0000 7f2356de9640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f233406c7d0 0x7f233406ec90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f233c036000 tx=0x7f233c03c040 comp rx=0 tx=0).stop 2023-10-30T17:32:01.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.430+0000 7f2356de9640 10 monclient: shutdown 2023-10-30T17:32:01.434 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.430+0000 7f2356de9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:01.434 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.430+0000 7f2356de9640 1 -- 172.21.15.170:0/2116114004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23500a9460 msgr2=0x7f23501425a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:01.434 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.430+0000 7f2356de9640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23500a9460 0x7f23501425a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f234400c490 tx=0x7f234400c950 comp rx=0 tx=0).stop 2023-10-30T17:32:01.434 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.431+0000 7f2356de9640 1 -- 172.21.15.170:0/2116114004 shutdown_connections 2023-10-30T17:32:01.435 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.431+0000 7f2356de9640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f233406c7d0 0x7f233406ec90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:01.435 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.431+0000 7f2356de9640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23501456a0 0x7f235013f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:01.435 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.431+0000 7f2356de9640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23500b4340 0x7f23501448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:01.435 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.431+0000 7f2356de9640 1 --2- 172.21.15.170:0/2116114004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23500a9460 0x7f23501425a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:01.435 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.431+0000 7f2356de9640 1 -- 172.21.15.170:0/2116114004 >> 172.21.15.170:0/2116114004 conn(0x7f2350071980 msgr2=0x7f235013dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:01.436 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.431+0000 7f2356de9640 1 -- 172.21.15.170:0/2116114004 shutdown_connections 2023-10-30T17:32:01.436 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.431+0000 7f2356de9640 1 -- 172.21.15.170:0/2116114004 wait complete. 2023-10-30T17:32:01.436 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:01.431+0000 7f2356de9640 1 librados: shutdown 2023-10-30T17:32:01.442 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T17:32:01.632 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:17.605 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:01.758 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:17.710 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:02.036 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:17.999 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:02.064 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:18.010 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:02.419 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:18.377 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:02.752 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:18.721 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:02.941 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:18.918 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:02.994 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:18.947 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:07.068 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:23.040 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:07.071 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:23.044 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:07.278 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:23.230 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:07.281 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:23.233 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:07.439 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:23.401 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:07.443 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:23.406 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:07.573 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:23.518 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:07.576 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:23.522 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:08.248 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T17:32:08.255 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:24.214 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:08.261 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:24.220 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:08.384 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.383+0000 7f9b8dbdc640 10 monclient: get_monmap_and_config 2023-10-30T17:32:08.385 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.383+0000 7f9b8dbdc640 10 monclient: build_initial_monmap 2023-10-30T17:32:08.385 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.383+0000 7f9b8dbdc640 10 monclient: monmap: 2023-10-30T17:32:08.385 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:32:08.385 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:08.386 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:32:08.384604+0000 2023-10-30T17:32:08.386 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:32:08.384604+0000 2023-10-30T17:32:08.386 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:08.386 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:08.386 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:08.387 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:08.387 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:08.387 INFO:teuthology.misc.health.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:08.387 INFO:teuthology.misc.health.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:08.387 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:08.389 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.384+0000 7f9b8dbdc640 1 Processor -- start 2023-10-30T17:32:08.389 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.384+0000 7f9b8dbdc640 1 -- start start 2023-10-30T17:32:08.389 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.384+0000 7f9b8dbdc640 10 monclient: init 2023-10-30T17:32:08.390 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.384+0000 7f9b8dbdc640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:08.390 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.384+0000 7f9b8dbdc640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-10-30T17:32:08.390 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.384+0000 7f9b8dbdc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b880b44b0 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:08.390 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.384+0000 7f9b8dbdc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9b8813f350 0x7f9b880a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:08.390 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.385+0000 7f9b8dbdc640 10 monclient(hunting): picked mon.noname-a con 0x7f9b8813f350 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:08.391 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.385+0000 7f9b8dbdc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b881419d0 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:08.391 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.385+0000 7f9b8dbdc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b88142590 con 0x7f9b880b44b0 2023-10-30T17:32:08.391 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.385+0000 7f9b8dbdc640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b880aa4b0 con 0x7f9b881419d0 2023-10-30T17:32:08.391 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.385+0000 7f9b8dbdc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:08.391 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.385+0000 7f9b8dbdc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b880b4890 con 0x7f9b8813f350 2023-10-30T17:32:08.392 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.385+0000 7f9b8dbdc640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:08.392 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.385+0000 7f9b8dbdc640 10 monclient(hunting): authenticate will time out at 1310.708127s 2023-10-30T17:32:08.392 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.385+0000 7f9b86ffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9b8813f350 0x7f9b880a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:08.392 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.386+0000 7f9b87fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f9b881419d0 0x7f9b88141e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:38358/0 (socket says 172.21.15.170:38358) 2023-10-30T17:32:08.392 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.386+0000 7f9b87fff640 1 -- 172.21.15.170:0/1985232874 learned_addr learned my addr 172.21.15.170:0/1985232874 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:08.393 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.386+0000 7f9b86ffd640 10 monclient(hunting): get_auth_request con 0x7f9b8813f350 auth_method 0 2023-10-30T17:32:08.393 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.386+0000 7f9b86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:08.393 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.386+0000 7f9b86ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:08.393 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.386+0000 7f9b86ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:08.393 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.387+0000 7f9b86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:08.394 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.387+0000 7f9b86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:08.394 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.387+0000 7f9b86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:08.395 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.389+0000 7f9b86ffd640 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2023-10-30T17:32:08.395 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.389+0000 7f9b86ffd640 1 -- 172.21.15.170:0/1985232874 >> v1:172.21.15.177:6789/0 conn(0x7f9b881419d0 legacy=0x7f9b88141e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:08.395 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.389+0000 7f9b86ffd640 1 -- 172.21.15.170:0/1985232874 >> v1:172.21.15.170:6789/0 conn(0x7f9b880b44b0 legacy=0x7f9b8813ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:08.395 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.389+0000 7f9b86ffd640 10 monclient: _finish_hunting 0 2023-10-30T17:32:08.395 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.389+0000 7f9b86ffd640 1 monclient: found mon.noname-a 2023-10-30T17:32:08.396 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.389+0000 7f9b86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:08.396 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.389+0000 7f9b86ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:08.396 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.389+0000 7f9b86ffd640 1 -- 172.21.15.170:0/1985232874 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b880aa790 con 0x7f9b8813f350 2023-10-30T17:32:08.396 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.392+0000 7f9b86ffd640 1 --2- 172.21.15.170:0/1985232874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 0x7f9b880a8b30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9b700097a0 tx=0x7f9b7002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fe0068bda4a553 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:08.397 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 1 -- 172.21.15.170:0/1985232874 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7002f810 con 0x7f9b8813f350 2023-10-30T17:32:08.398 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:08.398 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:08.398 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 10 monclient: dump: 2023-10-30T17:32:08.398 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:08.398 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:08.399 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:08.399 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:08.399 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:08.399 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:08.399 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:08.400 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:08.400 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:08.400 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:08.400 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 10 monclient: _finish_auth 0 2023-10-30T17:32:08.400 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 10 monclient: _check_auth_tickets 2023-10-30T17:32:08.401 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:08.401 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 1 -- 172.21.15.170:0/1985232874 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7002f970 con 0x7f9b8813f350 2023-10-30T17:32:08.401 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b8dbdc640 5 monclient: authenticate success, global_id 4227 2023-10-30T17:32:08.401 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b8dbdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:08.401 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:08.402 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b8dbdc640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:08.402 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b8dbdc640 4 set_mon_vals no callback set 2023-10-30T17:32:08.402 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 1 -- 172.21.15.170:0/1985232874 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7002fd00 con 0x7f9b8813f350 2023-10-30T17:32:08.402 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:08.402 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:08.402 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.396+0000 7f9b867fc640 10 monclient: dump: 2023-10-30T17:32:08.403 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:08.403 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:08.403 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:08.403 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:08.403 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:08.404 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:08.404 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:08.404 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:08.404 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:08.404 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:08.408 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.397+0000 7f9b8dbdc640 10 monclient: shutdown 2023-10-30T17:32:08.409 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.397+0000 7f9b8dbdc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:08.409 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.397+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/1985232874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 msgr2=0x7f9b880a8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:08.409 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.397+0000 7f9b8dbdc640 1 --2- 172.21.15.170:0/1985232874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 0x7f9b880a8b30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9b700097a0 tx=0x7f9b7002b9f0 comp rx=0 tx=0).stop 2023-10-30T17:32:08.409 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.397+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/1985232874 shutdown_connections 2023-10-30T17:32:08.409 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.397+0000 7f9b8dbdc640 1 --2- 172.21.15.170:0/1985232874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 0x7f9b880a8b30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:08.410 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.397+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/1985232874 >> 172.21.15.170:0/1985232874 conn(0x7f9b88071b60 msgr2=0x7f9b88071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:08.410 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.397+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/1985232874 shutdown_connections 2023-10-30T17:32:08.410 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.397+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/1985232874 wait complete. 2023-10-30T17:32:08.410 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.398+0000 7f9b8dbdc640 10 monclient: build_initial_monmap 2023-10-30T17:32:08.410 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.398+0000 7f9b8dbdc640 10 monclient: monmap: 2023-10-30T17:32:08.411 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:32:08.411 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:08.411 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:08.411 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2023-10-30T17:32:08.411 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:08.412 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:08.412 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:08.412 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:08.412 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:08.413 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:08.413 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.398+0000 7f9b8dbdc640 1 librados: starting msgr at 2023-10-30T17:32:08.414 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.398+0000 7f9b8dbdc640 1 librados: starting objecter 2023-10-30T17:32:08.414 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.398+0000 7f9b8dbdc640 1 Processor -- start 2023-10-30T17:32:08.414 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.398+0000 7f9b8dbdc640 1 -- start start 2023-10-30T17:32:08.414 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.398+0000 7f9b8dbdc640 1 librados: setting wanted keys 2023-10-30T17:32:08.414 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.398+0000 7f9b8dbdc640 1 librados: calling monclient init 2023-10-30T17:32:08.415 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.398+0000 7f9b8dbdc640 10 monclient: init 2023-10-30T17:32:08.415 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:08.415 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:32:08.415 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b880b44b0 0x7f9b88139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:08.415 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 10 monclient(hunting): picked mon.noname-c con 0x7f9b880b44b0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:08.416 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 0x7f9b881d6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:08.416 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 10 monclient(hunting): picked mon.noname-a con 0x7f9b8813f350 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:08.416 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b881419d0 0x7f9b881d7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:08.416 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 10 monclient(hunting): picked mon.noname-b con 0x7f9b881419d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:08.416 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:08.417 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b881d68c0 con 0x7f9b8813f350 2023-10-30T17:32:08.417 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:08.417 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b881382c0 con 0x7f9b881419d0 2023-10-30T17:32:08.417 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:08.417 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b88139c90 con 0x7f9b880b44b0 2023-10-30T17:32:08.418 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:08.418 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b8dbdc640 10 monclient(hunting): authenticate will time out at 1310.722434s 2023-10-30T17:32:08.418 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.399+0000 7f9b86ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 0x7f9b881d6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:08.418 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b87fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b881419d0 0x7f9b881d7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:08.418 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b877fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b880b44b0 0x7f9b88139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:08.419 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b86ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 0x7f9b881d6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:44824/0 (socket says 172.21.15.170:44824) 2023-10-30T17:32:08.419 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b86ffd640 1 -- 172.21.15.170:0/3511858350 learned_addr learned my addr 172.21.15.170:0/3511858350 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:08.419 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b86ffd640 10 monclient(hunting): get_auth_request con 0x7f9b8813f350 auth_method 0 2023-10-30T17:32:08.419 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:08.419 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b86ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:08.420 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b86ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:08.420 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b877fe640 10 monclient(hunting): get_auth_request con 0x7f9b880b44b0 auth_method 0 2023-10-30T17:32:08.420 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:08.420 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b877fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:08.420 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b877fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:08.421 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b87fff640 10 monclient(hunting): get_auth_request con 0x7f9b881419d0 auth_method 0 2023-10-30T17:32:08.421 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:08.421 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b87fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:08.421 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b87fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:08.421 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:08.422 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.400+0000 7f9b86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:08.422 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.401+0000 7f9b86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:08.422 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.401+0000 7f9b877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:08.422 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.401+0000 7f9b877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:08.422 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.401+0000 7f9b877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:08.423 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.401+0000 7f9b87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:08.423 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.401+0000 7f9b87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:08.423 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.401+0000 7f9b87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:08.423 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.403+0000 7f9b86ffd640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2023-10-30T17:32:08.423 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.403+0000 7f9b86ffd640 1 -- 172.21.15.170:0/3511858350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b880b44b0 msgr2=0x7f9b88139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:08.424 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.403+0000 7f9b86ffd640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b880b44b0 0x7f9b88139750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:08.424 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.403+0000 7f9b86ffd640 1 -- 172.21.15.170:0/3511858350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b881419d0 msgr2=0x7f9b881d7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:08.424 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.403+0000 7f9b86ffd640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b881419d0 0x7f9b881d7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:08.424 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.403+0000 7f9b86ffd640 10 monclient: _finish_hunting 0 2023-10-30T17:32:08.424 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.404+0000 7f9b86ffd640 1 monclient: found mon.noname-a 2023-10-30T17:32:08.425 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.404+0000 7f9b86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:08.425 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.404+0000 7f9b86ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:08.425 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.404+0000 7f9b86ffd640 1 -- 172.21.15.170:0/3511858350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b70009400 con 0x7f9b8813f350 2023-10-30T17:32:08.425 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.406+0000 7f9b86ffd640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 0x7f9b881d6d50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9b700098c0 tx=0x7f9b700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:08.425 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.406+0000 7f9b7ffff640 10 client.?.objecter ms_handle_connect 0x7f9b8813f350 2023-10-30T17:32:08.425 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.406+0000 7f9b7ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:08.427 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b7ffff640 1 -- 172.21.15.170:0/3511858350 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b70046070 con 0x7f9b8813f350 2023-10-30T17:32:08.428 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:08.428 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b7ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:08.428 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b7ffff640 10 monclient: dump: 2023-10-30T17:32:08.428 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:08.428 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:08.429 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:08.429 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:08.429 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:08.429 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:08.429 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:08.430 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:08.430 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:08.430 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:08.430 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b7ffff640 10 monclient: _finish_auth 0 2023-10-30T17:32:08.430 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b7ffff640 10 monclient: _check_auth_tickets 2023-10-30T17:32:08.431 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:08.431 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b8dbdc640 5 monclient: authenticate success, global_id 4230 2023-10-30T17:32:08.431 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b8dbdc640 10 monclient: _renew_subs 2023-10-30T17:32:08.431 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b8dbdc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:08.431 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/3511858350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b88139eb0 con 0x7f9b8813f350 2023-10-30T17:32:08.432 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b8dbdc640 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:08.432 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b8dbdc640 10 monclient: _renew_subs 2023-10-30T17:32:08.432 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b8dbdc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:08.432 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/3511858350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b88136810 con 0x7f9b8813f350 2023-10-30T17:32:08.432 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.409+0000 7f9b8dbdc640 1 librados: init done 2023-10-30T17:32:08.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.410+0000 7f9b8dbdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:08.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.410+0000 7f9b8dbdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:08.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.410+0000 7f9b8dbdc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:08.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.410+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/3511858350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b881d6a70 con 0x7f9b8813f350 2023-10-30T17:32:08.433 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.411+0000 7f9b7ffff640 1 -- 172.21.15.170:0/3511858350 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b70037850 con 0x7f9b8813f350 2023-10-30T17:32:08.434 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.411+0000 7f9b7ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:08.434 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.411+0000 7f9b7ffff640 1 -- 172.21.15.170:0/3511858350 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7003b040 con 0x7f9b8813f350 2023-10-30T17:32:08.434 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.411+0000 7f9b7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:08.434 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.411+0000 7f9b7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:08.435 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.411+0000 7f9b7ffff640 10 monclient: dump: 2023-10-30T17:32:08.435 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:08.435 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:08.435 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:08.435 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:08.436 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:08.436 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:08.436 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:08.436 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:08.436 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:08.437 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:08.437 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.411+0000 7f9b85ffb640 4 set_mon_vals no callback set 2023-10-30T17:32:08.677 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.676+0000 7f9b7ffff640 1 -- 172.21.15.170:0/3511858350 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f9b70037bc0 con 0x7f9b8813f350 2023-10-30T17:32:08.678 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.676+0000 7f9b7ffff640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9b6806c770 0x7f9b6806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:08.678 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.677+0000 7f9b877fe640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9b6806c770 0x7f9b6806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:08.679 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.677+0000 7f9b877fe640 10 monclient: get_auth_request con 0x7f9b6806c770 auth_method 0 2023-10-30T17:32:08.679 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.677+0000 7f9b877fe640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9b6806c770 0x7f9b6806ec30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9b78001f10 tx=0x7f9b78009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:08.679 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.677+0000 7f9b7ffff640 10 client.4230.objecter ms_handle_connect 0x7f9b6806c770 2023-10-30T17:32:08.709 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.707+0000 7f9b7ffff640 1 -- 172.21.15.170:0/3511858350 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9b7007b450 con 0x7f9b8813f350 2023-10-30T17:32:08.709 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.707+0000 7f9b7ffff640 10 client.4230.objecter ms_dispatch 0x7f9b88000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T17:32:08.710 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.707+0000 7f9b7ffff640 3 client.4230.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T17:32:08.710 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.707+0000 7f9b7ffff640 3 client.4230.objecter handle_osd_map decoding full epoch 10 2023-10-30T17:32:08.710 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.707+0000 7f9b7ffff640 20 client.4230.objecter dump_active .. 0 homeless 2023-10-30T17:32:08.710 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.707+0000 7f9b7ffff640 1 -- 172.21.15.170:0/3511858350 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9b70077790 con 0x7f9b8813f350 2023-10-30T17:32:08.711 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.708+0000 7f9b7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:08.711 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.708+0000 7f9b7ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:08.854 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:24.831 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:08.858 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:24.835 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:08.872 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:24.825 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:08.878 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:24.831 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:08.894 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.893+0000 7f9b8dbdc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:08.894 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.893+0000 7f9b8dbdc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:08.895 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.893+0000 7f9b8dbdc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:08.895 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.893+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/3511858350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b881d6a70 con 0x7f9b8813f350 2023-10-30T17:32:08.927 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.925+0000 7f9b7ffff640 1 -- 172.21.15.170:0/3511858350 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9b881d6a70 con 0x7f9b8813f350 2023-10-30T17:32:08.927 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.926+0000 7f9b7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:08.928 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.926+0000 7f9b7ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:08.928 INFO:teuthology.misc.health.smithi170.stdout: 2023-10-30T17:32:08.928 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:31:43.906314+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:32:08.930 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 10 librados: watch_flush enter 2023-10-30T17:32:08.930 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 10 librados: watch_flush exit 2023-10-30T17:32:08.930 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 20 client.4230.objecter shutdown clearing up homeless session... 2023-10-30T17:32:08.930 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 10 client.4230.objecter successfully canceled tick 2023-10-30T17:32:08.931 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/3511858350 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9b6806c770 msgr2=0x7f9b6806ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:08.931 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9b6806c770 0x7f9b6806ec30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9b78001f10 tx=0x7f9b78009040 comp rx=0 tx=0).stop 2023-10-30T17:32:08.931 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 10 monclient: shutdown 2023-10-30T17:32:08.931 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:08.931 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/3511858350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 msgr2=0x7f9b881d6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:08.932 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 0x7f9b881d6d50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9b700098c0 tx=0x7f9b700046c0 comp rx=0 tx=0).stop 2023-10-30T17:32:08.932 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/3511858350 shutdown_connections 2023-10-30T17:32:08.932 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9b6806c770 0x7f9b6806ec30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:08.932 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b881419d0 0x7f9b881d7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:08.932 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b8813f350 0x7f9b881d6d50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:08.932 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 --2- 172.21.15.170:0/3511858350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b880b44b0 0x7f9b88139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:08.933 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/3511858350 >> 172.21.15.170:0/3511858350 conn(0x7f9b88071b60 msgr2=0x7f9b88134b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:08.933 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/3511858350 shutdown_connections 2023-10-30T17:32:08.933 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 -- 172.21.15.170:0/3511858350 wait complete. 2023-10-30T17:32:08.933 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:08.928+0000 7f9b8dbdc640 1 librados: shutdown 2023-10-30T17:32:08.940 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T17:32:09.069 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:25.038 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:09.073 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:25.043 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:10.831 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:26.804 86233== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:11.363 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:27.315 86235== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:11.382 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:27.344 86234== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:11.744 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:27.689 86236== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:12.746 INFO:tasks.ceph.osd.3.smithi170.stderr:==00:00:00:28.705 86390== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:13.141 INFO:tasks.ceph.osd.2.smithi170.stderr:==00:00:00:29.095 86391== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:13.210 INFO:tasks.ceph.osd.1.smithi170.stderr:==00:00:00:29.187 86388== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:13.316 INFO:tasks.ceph.osd.0.smithi170.stderr:==00:00:00:29.285 86389== Warning: unimplemented fcntl command: 1038 2023-10-30T17:32:15.747 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T17:32:15.840 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.839+0000 7f279c2af640 10 monclient: get_monmap_and_config 2023-10-30T17:32:15.841 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.839+0000 7f279c2af640 10 monclient: build_initial_monmap 2023-10-30T17:32:15.841 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.839+0000 7f279c2af640 10 monclient: monmap: 2023-10-30T17:32:15.841 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:32:15.841 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:15.842 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:32:15.840294+0000 2023-10-30T17:32:15.842 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:32:15.840294+0000 2023-10-30T17:32:15.842 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:15.842 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:15.842 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:15.842 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:15.843 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:15.843 INFO:teuthology.misc.health.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:15.843 INFO:teuthology.misc.health.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:15.843 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:15.845 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.839+0000 7f279c2af640 1 Processor -- start 2023-10-30T17:32:15.845 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.840+0000 7f279c2af640 1 -- start start 2023-10-30T17:32:15.845 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.840+0000 7f279c2af640 10 monclient: init 2023-10-30T17:32:15.845 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.840+0000 7f279c2af640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:15.846 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.840+0000 7f279c2af640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-10-30T17:32:15.846 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.840+0000 7f279c2af640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f27940b4340 0x7f27940a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:15.846 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.840+0000 7f279c2af640 10 monclient(hunting): picked mon.noname-c con 0x7f27940b4340 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:15.846 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.840+0000 7f279c2af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27940a9460 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:15.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.840+0000 7f279c2af640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 0x7f279413e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:15.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.840+0000 7f279c2af640 10 monclient(hunting): picked mon.noname-b con 0x7f27941456a0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:15.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.841+0000 7f279c2af640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27940a8fd0 con 0x7f27940a9460 2023-10-30T17:32:15.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.841+0000 7f279c2af640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:15.847 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.841+0000 7f279c2af640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27940a8d20 con 0x7f27940b4340 2023-10-30T17:32:15.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.841+0000 7f279c2af640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:15.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.841+0000 7f279c2af640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27940aa400 con 0x7f27941456a0 2023-10-30T17:32:15.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.841+0000 7f279c2af640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:15.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.841+0000 7f279c2af640 10 monclient(hunting): authenticate will time out at 1318.164190s 2023-10-30T17:32:15.848 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.841+0000 7f279a04b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f27940b4340 0x7f27940a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:15.849 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.841+0000 7f279a84c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 0x7f279413e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:15.849 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a04b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f27940b4340 0x7f27940a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:45570/0 (socket says 172.21.15.170:45570) 2023-10-30T17:32:15.849 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a04b640 1 -- 172.21.15.170:0/2080273419 learned_addr learned my addr 172.21.15.170:0/2080273419 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:15.849 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a04b640 10 monclient(hunting): get_auth_request con 0x7f27940b4340 auth_method 0 2023-10-30T17:32:15.849 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:15.850 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a04b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:15.850 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a04b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:15.850 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a84c640 10 monclient(hunting): get_auth_request con 0x7f27941456a0 auth_method 0 2023-10-30T17:32:15.850 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a84c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:15.850 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a84c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:15.850 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a84c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:15.851 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a04b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:15.851 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a04b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:15.851 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.842+0000 7f279a04b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:15.851 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.843+0000 7f279a84c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:15.851 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.843+0000 7f279a84c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:15.852 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.843+0000 7f279a84c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:15.853 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.845+0000 7f279a04b640 10 monclient(hunting): handle_auth_done global_id 4222 payload 274 2023-10-30T17:32:15.853 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.845+0000 7f279a04b640 1 -- 172.21.15.170:0/2080273419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 msgr2=0x7f279413e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:15.854 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.845+0000 7f279a04b640 1 --2- 172.21.15.170:0/2080273419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 0x7f279413e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:15.854 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.845+0000 7f279a04b640 1 -- 172.21.15.170:0/2080273419 >> v1:172.21.15.170:6789/0 conn(0x7f27940a9460 legacy=0x7f2794144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:15.854 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.845+0000 7f279a04b640 10 monclient: _finish_hunting 0 2023-10-30T17:32:15.854 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.845+0000 7f279a04b640 1 monclient: found mon.noname-c 2023-10-30T17:32:15.854 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.845+0000 7f279a04b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:15.855 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.845+0000 7f279a04b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:32:15.855 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.845+0000 7f279a04b640 1 -- 172.21.15.170:0/2080273419 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27940aa6e0 con 0x7f27940b4340 2023-10-30T17:32:15.855 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.847+0000 7f279a04b640 1 --2- 172.21.15.170:0/2080273419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27940b4340 0x7f27940a87e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2790009870 tx=0x7f279002b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b94739b80122a26d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:15.855 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.850+0000 7f2799049640 1 -- 172.21.15.170:0/2080273419 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790031890 con 0x7f27940b4340 2023-10-30T17:32:15.855 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.850+0000 7f2799049640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:15.856 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.850+0000 7f2799049640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:15.856 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.850+0000 7f2799049640 10 monclient: dump: 2023-10-30T17:32:15.856 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:15.856 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:15.856 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:15.857 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:15.857 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:15.857 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:15.857 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:15.857 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:15.857 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:15.858 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:15.858 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.850+0000 7f2799049640 10 monclient: _finish_auth 0 2023-10-30T17:32:15.858 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.850+0000 7f2799049640 10 monclient: _check_auth_tickets 2023-10-30T17:32:15.858 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.850+0000 7f2799049640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:15.858 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.850+0000 7f279c2af640 5 monclient: authenticate success, global_id 4222 2023-10-30T17:32:15.859 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.850+0000 7f279c2af640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:15.863 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.852+0000 7f2799049640 1 -- 172.21.15.170:0/2080273419 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27900319f0 con 0x7f27940b4340 2023-10-30T17:32:15.864 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f2799049640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:15.864 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f2799049640 1 -- 172.21.15.170:0/2080273419 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790031dc0 con 0x7f27940b4340 2023-10-30T17:32:15.864 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f2799049640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:15.864 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f2799049640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:15.864 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f2799049640 10 monclient: dump: 2023-10-30T17:32:15.865 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:15.865 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:15.865 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:15.865 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:15.865 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:15.866 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:15.866 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:15.866 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:15.866 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:15.866 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:15.867 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f279c2af640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:15.867 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f279c2af640 4 set_mon_vals no callback set 2023-10-30T17:32:15.867 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f279c2af640 10 monclient: shutdown 2023-10-30T17:32:15.867 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f279c2af640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:15.867 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f279c2af640 1 -- 172.21.15.170:0/2080273419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27940b4340 msgr2=0x7f27940a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:15.868 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f279c2af640 1 --2- 172.21.15.170:0/2080273419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27940b4340 0x7f27940a87e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2790009870 tx=0x7f279002b930 comp rx=0 tx=0).stop 2023-10-30T17:32:15.868 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f279c2af640 1 -- 172.21.15.170:0/2080273419 shutdown_connections 2023-10-30T17:32:15.868 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f279c2af640 1 --2- 172.21.15.170:0/2080273419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 0x7f279413e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:15.868 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f279c2af640 1 --2- 172.21.15.170:0/2080273419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27940b4340 0x7f27940a87e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:15.868 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.853+0000 7f279c2af640 1 -- 172.21.15.170:0/2080273419 >> 172.21.15.170:0/2080273419 conn(0x7f2794071980 msgr2=0x7f2794071d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:15.869 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.854+0000 7f279c2af640 1 -- 172.21.15.170:0/2080273419 shutdown_connections 2023-10-30T17:32:15.869 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.854+0000 7f279c2af640 1 -- 172.21.15.170:0/2080273419 wait complete. 2023-10-30T17:32:15.869 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.854+0000 7f279c2af640 10 monclient: build_initial_monmap 2023-10-30T17:32:15.869 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.854+0000 7f279c2af640 10 monclient: monmap: 2023-10-30T17:32:15.869 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:32:15.870 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:15.870 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:15.870 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2023-10-30T17:32:15.870 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:15.870 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:15.871 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:15.871 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:15.871 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:15.871 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:15.871 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.855+0000 7f279c2af640 1 librados: starting msgr at 2023-10-30T17:32:15.872 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.855+0000 7f279c2af640 1 librados: starting objecter 2023-10-30T17:32:15.872 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.855+0000 7f279c2af640 1 Processor -- start 2023-10-30T17:32:15.872 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.855+0000 7f279c2af640 1 -- start start 2023-10-30T17:32:15.872 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.855+0000 7f279c2af640 1 librados: setting wanted keys 2023-10-30T17:32:15.872 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.855+0000 7f279c2af640 1 librados: calling monclient init 2023-10-30T17:32:15.873 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.855+0000 7f279c2af640 10 monclient: init 2023-10-30T17:32:15.873 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.856+0000 7f279c2af640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:15.873 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.856+0000 7f279c2af640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:32:15.873 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.856+0000 7f279c2af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27940a9460 0x7f27940abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:15.873 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.856+0000 7f279c2af640 10 monclient(hunting): picked mon.noname-b con 0x7f27940a9460 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:15.874 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.856+0000 7f279c2af640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27940b4340 0x7f2794138770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:15.874 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.856+0000 7f279c2af640 10 monclient(hunting): picked mon.noname-a con 0x7f27940b4340 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:15.874 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.856+0000 7f279c2af640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 0x7f2794138cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:15.874 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.857+0000 7f279984a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27940b4340 0x7f2794138770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:15.874 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.857+0000 7f279a04b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27940a9460 0x7f27940abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:15.875 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.857+0000 7f279a04b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27940a9460 0x7f27940abf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:45584/0 (socket says 172.21.15.170:45584) 2023-10-30T17:32:15.875 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.857+0000 7f279984a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27940b4340 0x7f2794138770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:35494/0 (socket says 172.21.15.170:35494) 2023-10-30T17:32:15.875 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a84c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 0x7f2794138cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:15.875 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a84c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 0x7f2794138cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:35834/0 (socket says 172.21.15.170:35834) 2023-10-30T17:32:15.875 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279c2af640 10 monclient(hunting): picked mon.noname-c con 0x7f27941456a0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:15.876 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279c2af640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:15.876 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a04b640 1 -- 172.21.15.170:0/3137914143 learned_addr learned my addr 172.21.15.170:0/3137914143 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:15.876 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27941391f0 con 0x7f27940b4340 2023-10-30T17:32:15.876 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279c2af640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:15.876 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27940ad870 con 0x7f27940a9460 2023-10-30T17:32:15.877 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279c2af640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:15.877 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27941405b0 con 0x7f27941456a0 2023-10-30T17:32:15.877 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279c2af640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:15.877 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279c2af640 10 monclient(hunting): authenticate will time out at 1318.181794s 2023-10-30T17:32:15.877 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a04b640 10 monclient(hunting): get_auth_request con 0x7f27940a9460 auth_method 0 2023-10-30T17:32:15.877 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:15.878 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a04b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:15.878 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a04b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:15.878 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279984a640 10 monclient(hunting): get_auth_request con 0x7f27940b4340 auth_method 0 2023-10-30T17:32:15.878 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279984a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:15.878 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279984a640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:15.879 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279984a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:15.879 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a84c640 10 monclient(hunting): get_auth_request con 0x7f27941456a0 auth_method 0 2023-10-30T17:32:15.879 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a84c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:15.879 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a84c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:15.879 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.858+0000 7f279a84c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:15.880 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.859+0000 7f279a04b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:15.880 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.859+0000 7f279a04b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:15.880 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.859+0000 7f279a04b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:15.880 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.859+0000 7f279984a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:15.880 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.859+0000 7f279984a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:15.881 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.859+0000 7f279984a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:15.881 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.859+0000 7f279a84c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:15.881 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.859+0000 7f279a84c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:15.881 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.859+0000 7f279a84c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:15.884 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.863+0000 7f279984a640 10 monclient(hunting): handle_auth_done global_id 4236 payload 824 2023-10-30T17:32:15.884 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.863+0000 7f279984a640 1 -- 172.21.15.170:0/3137914143 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 msgr2=0x7f2794138cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:15.884 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.863+0000 7f279984a640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 0x7f2794138cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:15.885 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.863+0000 7f279984a640 1 -- 172.21.15.170:0/3137914143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27940a9460 msgr2=0x7f27940abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:15.885 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.863+0000 7f279984a640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27940a9460 0x7f27940abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:15.885 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.863+0000 7f279984a640 10 monclient: _finish_hunting 0 2023-10-30T17:32:15.885 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.863+0000 7f279984a640 1 monclient: found mon.noname-a 2023-10-30T17:32:15.885 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.863+0000 7f279984a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:15.886 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.863+0000 7f279984a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:15.886 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.863+0000 7f279984a640 1 -- 172.21.15.170:0/3137914143 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27900094d0 con 0x7f27940b4340 2023-10-30T17:32:15.886 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.865+0000 7f279984a640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27940b4340 0x7f2794138770 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f278400c390 tx=0x7f278400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:15.886 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.866+0000 7f278affd640 10 client.?.objecter ms_handle_connect 0x7f27940b4340 2023-10-30T17:32:15.887 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.866+0000 7f278affd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:15.887 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 1 -- 172.21.15.170:0/3137914143 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27840052d0 con 0x7f27940b4340 2023-10-30T17:32:15.887 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:15.887 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:15.887 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 10 monclient: dump: 2023-10-30T17:32:15.888 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:15.888 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:15.888 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:15.888 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:15.888 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:15.889 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:15.889 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:15.889 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:15.889 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:15.889 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:15.890 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 10 monclient: _finish_auth 0 2023-10-30T17:32:15.890 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:15.890 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:15.890 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 1 -- 172.21.15.170:0/3137914143 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2784005430 con 0x7f27940b4340 2023-10-30T17:32:15.890 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f279c2af640 5 monclient: authenticate success, global_id 4236 2023-10-30T17:32:15.891 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f279c2af640 10 monclient: _renew_subs 2023-10-30T17:32:15.891 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f279c2af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:15.891 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27940ac490 con 0x7f27940b4340 2023-10-30T17:32:15.891 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:15.891 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f279c2af640 10 client.4236.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:15.892 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f279c2af640 10 monclient: _renew_subs 2023-10-30T17:32:15.892 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f279c2af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:15.892 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 1 -- 172.21.15.170:0/3137914143 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784003830 con 0x7f27940b4340 2023-10-30T17:32:15.892 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27941d4400 con 0x7f27940b4340 2023-10-30T17:32:15.892 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:15.893 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:15.893 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f2798848640 4 set_mon_vals no callback set 2023-10-30T17:32:15.893 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f278affd640 10 monclient: dump: 2023-10-30T17:32:15.893 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:15.894 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:15.894 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:15.894 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:15.894 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:15.894 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:15.895 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:15.895 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:15.895 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:15.895 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:15.895 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.869+0000 7f279c2af640 1 librados: init done 2023-10-30T17:32:15.896 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.871+0000 7f279c2af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:15.896 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.871+0000 7f279c2af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:15.896 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.871+0000 7f279c2af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:15.896 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.871+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f279404e3c0 con 0x7f27940b4340 2023-10-30T17:32:15.897 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.895+0000 7f278affd640 1 -- 172.21.15.170:0/3137914143 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f2784003990 con 0x7f27940b4340 2023-10-30T17:32:15.897 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.895+0000 7f278affd640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f277806c770 0x7f277806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:15.898 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.895+0000 7f279a04b640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f277806c770 0x7f277806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:15.898 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.896+0000 7f279a04b640 10 monclient: get_auth_request con 0x7f277806c770 auth_method 0 2023-10-30T17:32:15.898 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.896+0000 7f278affd640 10 client.4236.objecter ms_handle_connect 0x7f277806c770 2023-10-30T17:32:15.898 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.896+0000 7f279a04b640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f277806c770 0x7f277806ec30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2790009870 tx=0x7f279003c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:15.899 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.897+0000 7f278affd640 1 -- 172.21.15.170:0/3137914143 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2784057dc0 con 0x7f27940b4340 2023-10-30T17:32:15.899 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.897+0000 7f278affd640 10 client.4236.objecter ms_dispatch 0x7f2794000b90 osd_map(10..10 src has 1..10) v4 2023-10-30T17:32:15.899 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.897+0000 7f278affd640 3 client.4236.objecter handle_osd_map got epochs [10,10] > 0 2023-10-30T17:32:15.900 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.897+0000 7f278affd640 3 client.4236.objecter handle_osd_map decoding full epoch 10 2023-10-30T17:32:15.900 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:15.897+0000 7f278affd640 20 client.4236.objecter dump_active .. 0 homeless 2023-10-30T17:32:16.287 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.286+0000 7f278affd640 1 -- 172.21.15.170:0/3137914143 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2784054100 con 0x7f27940b4340 2023-10-30T17:32:16.288 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.286+0000 7f278affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:16.288 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.286+0000 7f278affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:16.495 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.494+0000 7f279c2af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:16.495 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.494+0000 7f279c2af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:16.495 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.494+0000 7f279c2af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:16.496 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.494+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27940b5470 con 0x7f27940b4340 2023-10-30T17:32:16.510 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.509+0000 7f278affd640 1 -- 172.21.15.170:0/3137914143 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2784017090 con 0x7f27940b4340 2023-10-30T17:32:16.511 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.509+0000 7f278affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:16.511 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.509+0000 7f278affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:16.511 INFO:teuthology.misc.health.smithi170.stdout: 2023-10-30T17:32:16.512 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:31:43.906314+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:32:16.513 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.511+0000 7f279c2af640 10 librados: watch_flush enter 2023-10-30T17:32:16.513 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.511+0000 7f279c2af640 10 librados: watch_flush exit 2023-10-30T17:32:16.513 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.511+0000 7f279c2af640 20 client.4236.objecter shutdown clearing up homeless session... 2023-10-30T17:32:16.513 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.511+0000 7f279c2af640 10 client.4236.objecter successfully canceled tick 2023-10-30T17:32:16.514 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.511+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f277806c770 msgr2=0x7f277806ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:16.514 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.511+0000 7f279c2af640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f277806c770 0x7f277806ec30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2790009870 tx=0x7f279003c040 comp rx=0 tx=0).stop 2023-10-30T17:32:16.514 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.511+0000 7f279c2af640 10 monclient: shutdown 2023-10-30T17:32:16.514 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.511+0000 7f279c2af640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:16.514 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.511+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27940b4340 msgr2=0x7f2794138770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:16.515 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.511+0000 7f279c2af640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27940b4340 0x7f2794138770 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f278400c390 tx=0x7f278400c850 comp rx=0 tx=0).stop 2023-10-30T17:32:16.515 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.512+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 shutdown_connections 2023-10-30T17:32:16.515 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.512+0000 7f279c2af640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f277806c770 0x7f277806ec30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:16.515 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.512+0000 7f279c2af640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27941456a0 0x7f2794138cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:16.515 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.512+0000 7f279c2af640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27940b4340 0x7f2794138770 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:16.516 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.512+0000 7f279c2af640 1 --2- 172.21.15.170:0/3137914143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27940a9460 0x7f27940abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:16.516 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.512+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 >> 172.21.15.170:0/3137914143 conn(0x7f2794071980 msgr2=0x7f2794071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:16.526 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.524+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 shutdown_connections 2023-10-30T17:32:16.526 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.525+0000 7f279c2af640 1 -- 172.21.15.170:0/3137914143 wait complete. 2023-10-30T17:32:16.526 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:16.525+0000 7f279c2af640 1 librados: shutdown 2023-10-30T17:32:16.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T17:32:19.173 INFO:tasks.ceph.osd.4.smithi177.stderr:2023-10-30T17:32:19.166+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-10-30T17:32:19.345 INFO:tasks.ceph.osd.5.smithi177.stderr:2023-10-30T17:32:19.341+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-10-30T17:32:19.359 INFO:tasks.ceph.osd.6.smithi177.stderr:2023-10-30T17:32:19.294+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-10-30T17:32:20.279 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:32:20.274+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-10-30T17:32:21.145 INFO:tasks.ceph.osd.2.smithi170.stderr:2023-10-30T17:32:21.141+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-10-30T17:32:21.586 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-30T17:32:21.493+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-10-30T17:32:21.989 INFO:tasks.ceph.osd.1.smithi170.stderr:2023-10-30T17:32:21.986+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-10-30T17:32:22.108 INFO:tasks.ceph.osd.0.smithi170.stderr:2023-10-30T17:32:22.104+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-10-30T17:32:22.666 INFO:tasks.ceph.osd.5.smithi177.stderr:2023-10-30T17:32:22.660+0000 ac7d640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T17:32:22.670 INFO:tasks.ceph.osd.4.smithi177.stderr:2023-10-30T17:32:22.663+0000 c495640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T17:32:22.731 INFO:tasks.ceph.osd.6.smithi177.stderr:2023-10-30T17:32:22.726+0000 c495640 -1 osd.6 0 waiting for initial osdmap 2023-10-30T17:32:23.100 INFO:tasks.ceph.osd.4.smithi177.stderr:2023-10-30T17:32:23.086+0000 11a5e640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T17:32:23.109 INFO:tasks.ceph.osd.5.smithi177.stderr:2023-10-30T17:32:23.086+0000 11a5e640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T17:32:23.129 INFO:tasks.ceph.osd.6.smithi177.stderr:2023-10-30T17:32:23.114+0000 11a5e640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T17:32:23.346 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T17:32:23.463 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.461+0000 7fb23a839640 10 monclient: get_monmap_and_config 2023-10-30T17:32:23.463 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.461+0000 7fb23a839640 10 monclient: build_initial_monmap 2023-10-30T17:32:23.463 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.461+0000 7fb23a839640 10 monclient: monmap: 2023-10-30T17:32:23.463 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:32:23.464 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:23.464 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:32:23.462784+0000 2023-10-30T17:32:23.464 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:32:23.462784+0000 2023-10-30T17:32:23.464 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:23.464 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:23.465 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:23.465 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:23.465 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:23.465 INFO:teuthology.misc.health.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:23.465 INFO:teuthology.misc.health.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:23.466 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:23.468 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.462+0000 7fb23a839640 1 Processor -- start 2023-10-30T17:32:23.468 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.462+0000 7fb23a839640 1 -- start start 2023-10-30T17:32:23.468 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.462+0000 7fb23a839640 10 monclient: init 2023-10-30T17:32:23.468 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:23.469 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-10-30T17:32:23.469 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2340b44b0 0x7fb2340a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:23.469 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 10 monclient(hunting): picked mon.noname-b con 0x7fb2340b44b0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:23.469 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb23413ec40 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:23.469 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb2341419d0 0x7fb234141e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:23.470 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 10 monclient(hunting): picked mon.noname-a con 0x7fb2341419d0 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:23.470 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2340aa4b0 con 0x7fb23413ec40 2023-10-30T17:32:23.470 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:23.470 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2340b4890 con 0x7fb2341419d0 2023-10-30T17:32:23.470 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:23.471 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340aa790 con 0x7fb2340b44b0 2023-10-30T17:32:23.471 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:23.471 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb23a839640 10 monclient(hunting): authenticate will time out at 1325.786368s 2023-10-30T17:32:23.471 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb233fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2340b44b0 0x7fb2340a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:23.471 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb2337fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb23413ec40 0x7fb23413f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:52440/0 (socket says 172.21.15.170:52440) 2023-10-30T17:32:23.471 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.463+0000 7fb2337fe640 1 -- 172.21.15.170:0/3132739896 learned_addr learned my addr 172.21.15.170:0/3132739896 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:23.472 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.464+0000 7fb233fff640 10 monclient(hunting): get_auth_request con 0x7fb2340b44b0 auth_method 0 2023-10-30T17:32:23.472 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.464+0000 7fb233fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:23.472 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.464+0000 7fb233fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:23.472 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.464+0000 7fb233fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:23.472 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.464+0000 7fb233fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:23.473 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.464+0000 7fb233fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:23.473 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.464+0000 7fb233fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:23.473 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.466+0000 7fb233fff640 10 monclient(hunting): handle_auth_done global_id 4217 payload 274 2023-10-30T17:32:23.473 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.466+0000 7fb233fff640 1 -- 172.21.15.170:0/3132739896 >> v2:172.21.15.170:3300/0 conn(0x7fb2341419d0 msgr2=0x7fb234141e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:23.473 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.466+0000 7fb233fff640 1 --2- 172.21.15.170:0/3132739896 >> v2:172.21.15.170:3300/0 conn(0x7fb2341419d0 0x7fb234141e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:23.474 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.466+0000 7fb233fff640 1 -- 172.21.15.170:0/3132739896 >> v1:172.21.15.177:6789/0 conn(0x7fb23413ec40 legacy=0x7fb23413f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:23.474 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.466+0000 7fb233fff640 10 monclient: _finish_hunting 0 2023-10-30T17:32:23.474 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.466+0000 7fb233fff640 1 monclient: found mon.noname-b 2023-10-30T17:32:23.474 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.466+0000 7fb233fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:23.474 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.466+0000 7fb233fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:23.474 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.466+0000 7fb233fff640 1 -- 172.21.15.170:0/3132739896 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2340a9200 con 0x7fb2340b44b0 2023-10-30T17:32:23.477 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.468+0000 7fb233fff640 1 --2- 172.21.15.170:0/3132739896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2340b44b0 0x7fb2340a8b30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2240097a0 tx=0x7fb22402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6184234c4e43c43d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:23.478 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 1 -- 172.21.15.170:0/3132739896 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224030610 con 0x7fb2340b44b0 2023-10-30T17:32:23.478 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:23.478 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:23.478 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 10 monclient: dump: 2023-10-30T17:32:23.478 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:23.479 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:23.479 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:23.479 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:23.479 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:23.480 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:23.480 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:23.480 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:23.480 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:23.480 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:23.481 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 10 monclient: _finish_auth 0 2023-10-30T17:32:23.481 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:23.481 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:23.481 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 1 -- 172.21.15.170:0/3132739896 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb224030770 con 0x7fb2340b44b0 2023-10-30T17:32:23.481 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb23a839640 5 monclient: authenticate success, global_id 4217 2023-10-30T17:32:23.481 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb23a839640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:23.482 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:23.482 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb23a839640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:23.482 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb23a839640 4 set_mon_vals no callback set 2023-10-30T17:32:23.482 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 1 -- 172.21.15.170:0/3132739896 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224030b00 con 0x7fb2340b44b0 2023-10-30T17:32:23.482 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:23.483 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:23.483 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.471+0000 7fb232ffd640 10 monclient: dump: 2023-10-30T17:32:23.483 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:23.483 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:23.483 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:23.483 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:23.484 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:23.484 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:23.484 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:23.484 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:23.484 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:23.485 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:23.485 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.472+0000 7fb23a839640 10 monclient: shutdown 2023-10-30T17:32:23.485 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.472+0000 7fb23a839640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:23.485 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.472+0000 7fb23a839640 1 -- 172.21.15.170:0/3132739896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2340b44b0 msgr2=0x7fb2340a8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:23.485 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.472+0000 7fb23a839640 1 --2- 172.21.15.170:0/3132739896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2340b44b0 0x7fb2340a8b30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2240097a0 tx=0x7fb22402b960 comp rx=0 tx=0).stop 2023-10-30T17:32:23.486 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.472+0000 7fb23a839640 1 -- 172.21.15.170:0/3132739896 shutdown_connections 2023-10-30T17:32:23.486 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.472+0000 7fb23a839640 1 --2- 172.21.15.170:0/3132739896 >> v2:172.21.15.170:3300/0 conn(0x7fb2341419d0 0x7fb234141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:23.486 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.472+0000 7fb23a839640 1 --2- 172.21.15.170:0/3132739896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2340b44b0 0x7fb2340a8b30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:23.486 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.472+0000 7fb23a839640 1 -- 172.21.15.170:0/3132739896 >> 172.21.15.170:0/3132739896 conn(0x7fb234071b60 msgr2=0x7fb234071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:23.486 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.472+0000 7fb23a839640 1 -- 172.21.15.170:0/3132739896 shutdown_connections 2023-10-30T17:32:23.486 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.472+0000 7fb23a839640 1 -- 172.21.15.170:0/3132739896 wait complete. 2023-10-30T17:32:23.487 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.473+0000 7fb23a839640 10 monclient: build_initial_monmap 2023-10-30T17:32:23.487 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.473+0000 7fb23a839640 10 monclient: monmap: 2023-10-30T17:32:23.487 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:32:23.487 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:23.487 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:23.488 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2023-10-30T17:32:23.488 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:23.488 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:23.488 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:23.488 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:23.488 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:23.489 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:23.489 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.473+0000 7fb23a839640 1 librados: starting msgr at 2023-10-30T17:32:23.489 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.473+0000 7fb23a839640 1 librados: starting objecter 2023-10-30T17:32:23.489 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.474+0000 7fb23a839640 1 Processor -- start 2023-10-30T17:32:23.489 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.474+0000 7fb23a839640 1 -- start start 2023-10-30T17:32:23.490 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.474+0000 7fb23a839640 1 librados: setting wanted keys 2023-10-30T17:32:23.490 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.474+0000 7fb23a839640 1 librados: calling monclient init 2023-10-30T17:32:23.490 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.474+0000 7fb23a839640 10 monclient: init 2023-10-30T17:32:23.490 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:23.490 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:32:23.491 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2340b44b0 0x7fb234139460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:23.491 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 10 monclient(hunting): picked mon.noname-a con 0x7fb2340b44b0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:23.491 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb23413ec40 0x7fb234144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:23.491 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 10 monclient(hunting): picked mon.noname-c con 0x7fb23413ec40 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:23.491 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2341419d0 0x7fb2341d6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:23.491 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 10 monclient(hunting): picked mon.noname-b con 0x7fb2341419d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:23.492 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:23.492 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb234145190 con 0x7fb2340b44b0 2023-10-30T17:32:23.492 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:23.492 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb234144e90 con 0x7fb2341419d0 2023-10-30T17:32:23.492 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:23.493 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb234145620 con 0x7fb23413ec40 2023-10-30T17:32:23.493 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:23.493 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.475+0000 7fb23a839640 10 monclient(hunting): authenticate will time out at 1325.798386s 2023-10-30T17:32:23.495 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.478+0000 7fb233fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2340b44b0 0x7fb234139460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:23.496 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.478+0000 7fb233fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2340b44b0 0x7fb234139460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:44654/0 (socket says 172.21.15.170:44654) 2023-10-30T17:32:23.496 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.478+0000 7fb233fff640 1 -- 172.21.15.170:0/2059660681 learned_addr learned my addr 172.21.15.170:0/2059660681 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:23.496 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.478+0000 7fb233fff640 10 monclient(hunting): get_auth_request con 0x7fb2340b44b0 auth_method 0 2023-10-30T17:32:23.496 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.478+0000 7fb233fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:23.496 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.478+0000 7fb233fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:23.497 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.478+0000 7fb233fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:23.497 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.479+0000 7fb233fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:23.497 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.479+0000 7fb233fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:23.497 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.479+0000 7fb233fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:23.497 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.481+0000 7fb233fff640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2023-10-30T17:32:23.497 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.481+0000 7fb233fff640 1 -- 172.21.15.170:0/2059660681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb23413ec40 msgr2=0x7fb234144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:23.498 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.481+0000 7fb233fff640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb23413ec40 0x7fb234144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:23.498 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.481+0000 7fb233fff640 1 -- 172.21.15.170:0/2059660681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2341419d0 msgr2=0x7fb2341d6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:23.498 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.481+0000 7fb233fff640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2341419d0 0x7fb2341d6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:23.498 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.481+0000 7fb233fff640 10 monclient: _finish_hunting 0 2023-10-30T17:32:23.498 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.481+0000 7fb233fff640 1 monclient: found mon.noname-a 2023-10-30T17:32:23.499 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.481+0000 7fb233fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:23.499 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.481+0000 7fb233fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:23.499 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.481+0000 7fb233fff640 1 -- 172.21.15.170:0/2059660681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb224009400 con 0x7fb2340b44b0 2023-10-30T17:32:23.499 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.484+0000 7fb233fff640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2340b44b0 0x7fb234139460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb22402bf70 tx=0x7fb224004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:23.499 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.484+0000 7fb230ff9640 10 client.?.objecter ms_handle_connect 0x7fb2340b44b0 2023-10-30T17:32:23.500 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.484+0000 7fb230ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:23.500 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 1 -- 172.21.15.170:0/2059660681 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22403e040 con 0x7fb2340b44b0 2023-10-30T17:32:23.500 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:23.500 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:23.500 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 10 monclient: dump: 2023-10-30T17:32:23.501 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:23.501 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:23.501 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:23.501 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:23.501 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:23.501 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:23.502 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:23.502 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:23.502 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:23.502 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:23.502 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 10 monclient: _finish_auth 0 2023-10-30T17:32:23.503 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 10 monclient: _check_auth_tickets 2023-10-30T17:32:23.503 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:23.503 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 1 -- 172.21.15.170:0/2059660681 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb224039cb0 con 0x7fb2340b44b0 2023-10-30T17:32:23.503 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb23a839640 5 monclient: authenticate success, global_id 4251 2023-10-30T17:32:23.503 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb23a839640 10 monclient: _renew_subs 2023-10-30T17:32:23.503 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb23a839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:23.504 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb23a839640 1 -- 172.21.15.170:0/2059660681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb234144040 con 0x7fb2340b44b0 2023-10-30T17:32:23.504 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:23.504 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 1 -- 172.21.15.170:0/2059660681 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224043680 con 0x7fb2340b44b0 2023-10-30T17:32:23.504 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:23.504 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:23.505 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb230ff9640 10 monclient: dump: 2023-10-30T17:32:23.505 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:23.505 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:23.505 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:23.505 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:23.505 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:23.506 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:23.506 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:23.506 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:23.506 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:23.506 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:23.507 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb2327fc640 4 set_mon_vals no callback set 2023-10-30T17:32:23.507 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb23a839640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:23.507 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb23a839640 10 monclient: _renew_subs 2023-10-30T17:32:23.507 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb23a839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:23.507 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb23a839640 1 -- 172.21.15.170:0/2059660681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2341d6e80 con 0x7fb2340b44b0 2023-10-30T17:32:23.508 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.487+0000 7fb23a839640 1 librados: init done 2023-10-30T17:32:23.508 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.488+0000 7fb23a839640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:23.508 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.488+0000 7fb23a839640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:23.508 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.488+0000 7fb23a839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:23.508 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.488+0000 7fb23a839640 1 -- 172.21.15.170:0/2059660681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb234067cd0 con 0x7fb2340b44b0 2023-10-30T17:32:23.509 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.507+0000 7fb230ff9640 1 -- 172.21.15.170:0/2059660681 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fb2240423e0 con 0x7fb2340b44b0 2023-10-30T17:32:23.509 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.508+0000 7fb230ff9640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fb22006c7c0 0x7fb22006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:23.510 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.508+0000 7fb2337fe640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fb22006c7c0 0x7fb22006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:23.510 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.508+0000 7fb2337fe640 10 monclient: get_auth_request con 0x7fb22006c7c0 auth_method 0 2023-10-30T17:32:23.510 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.508+0000 7fb230ff9640 10 client.4251.objecter ms_handle_connect 0x7fb22006c7c0 2023-10-30T17:32:23.510 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.508+0000 7fb2337fe640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fb22006c7c0 0x7fb22006ec80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb2341d4530 tx=0x7fb228007a40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:23.745 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.744+0000 7fb230ff9640 1 -- 172.21.15.170:0/2059660681 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 1957+0+0 (secure 0 0 0) 0x7fb22407b3d0 con 0x7fb2340b44b0 2023-10-30T17:32:23.746 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.744+0000 7fb230ff9640 10 client.4251.objecter ms_dispatch 0x7fb234000b90 osd_map(12..12 src has 1..12) v4 2023-10-30T17:32:23.746 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.744+0000 7fb230ff9640 3 client.4251.objecter handle_osd_map got epochs [12,12] > 0 2023-10-30T17:32:23.746 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.744+0000 7fb230ff9640 3 client.4251.objecter handle_osd_map decoding full epoch 12 2023-10-30T17:32:23.746 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.744+0000 7fb230ff9640 20 client.4251.objecter dump_active .. 0 homeless 2023-10-30T17:32:23.747 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.744+0000 7fb230ff9640 1 -- 172.21.15.170:0/2059660681 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb224077010 con 0x7fb2340b44b0 2023-10-30T17:32:23.747 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.744+0000 7fb230ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:23.747 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.744+0000 7fb230ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:23.950 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.949+0000 7fb23a839640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:23.951 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.949+0000 7fb23a839640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:23.951 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.949+0000 7fb23a839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:23.951 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.949+0000 7fb23a839640 1 -- 172.21.15.170:0/2059660681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb23413df80 con 0x7fb2340b44b0 2023-10-30T17:32:23.992 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.990+0000 7fb230ff9640 1 -- 172.21.15.170:0/2059660681 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+7566 (secure 0 0 0) 0x7fb224048090 con 0x7fb2340b44b0 2023-10-30T17:32:23.992 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.990+0000 7fb230ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:23.992 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.991+0000 7fb230ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:23.993 INFO:teuthology.misc.health.smithi170.stdout: 2023-10-30T17:32:23.993 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":13,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:32:23.380652+0000","last_up_change":"2023-10-30T17:32:23.380652+0000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6810","nonce":2243649533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6813","nonce":2243649533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6817","nonce":2243649533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6815","nonce":2243649533}]},"public_addr":"172.21.15.177:6810/2243649533","cluster_addr":"172.21.15.177:6813/2243649533","heartbeat_back_addr":"172.21.15.177:6817/2243649533","heartbeat_front_addr":"172.21.15.177:6815/2243649533","state":["exists","up"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:32:23.995 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.992+0000 7fb23a839640 10 librados: watch_flush enter 2023-10-30T17:32:23.995 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 10 librados: watch_flush exit 2023-10-30T17:32:23.995 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 20 client.4251.objecter shutdown clearing up homeless session... 2023-10-30T17:32:23.995 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 10 client.4251.objecter successfully canceled tick 2023-10-30T17:32:23.995 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 -- 172.21.15.170:0/2059660681 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fb22006c7c0 msgr2=0x7fb22006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:23.996 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fb22006c7c0 0x7fb22006ec80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb2341d4530 tx=0x7fb228007a40 comp rx=0 tx=0).stop 2023-10-30T17:32:23.996 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 10 monclient: shutdown 2023-10-30T17:32:23.996 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:23.996 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 -- 172.21.15.170:0/2059660681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2340b44b0 msgr2=0x7fb234139460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:23.997 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2340b44b0 0x7fb234139460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb22402bf70 tx=0x7fb224004730 comp rx=0 tx=0).stop 2023-10-30T17:32:23.997 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 -- 172.21.15.170:0/2059660681 shutdown_connections 2023-10-30T17:32:23.997 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fb22006c7c0 0x7fb22006ec80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:23.997 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2341419d0 0x7fb2341d6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:23.997 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb23413ec40 0x7fb234144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:23.998 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 --2- 172.21.15.170:0/2059660681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2340b44b0 0x7fb234139460 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:23.998 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 -- 172.21.15.170:0/2059660681 >> 172.21.15.170:0/2059660681 conn(0x7fb234071b60 msgr2=0x7fb2340b2380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:23.998 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 -- 172.21.15.170:0/2059660681 shutdown_connections 2023-10-30T17:32:23.998 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 -- 172.21.15.170:0/2059660681 wait complete. 2023-10-30T17:32:23.998 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:23.993+0000 7fb23a839640 1 librados: shutdown 2023-10-30T17:32:24.003 DEBUG:teuthology.misc:1 of 8 OSDs are up 2023-10-30T17:32:24.075 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:32:24.070+0000 c495640 -1 osd.7 0 waiting for initial osdmap 2023-10-30T17:32:24.452 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:32:24.438+0000 1225f640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T17:32:25.173 INFO:tasks.ceph.osd.2.smithi170.stderr:2023-10-30T17:32:25.167+0000 c495640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T17:32:25.210 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-30T17:32:25.199+0000 c495640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T17:32:25.222 INFO:tasks.ceph.osd.1.smithi170.stderr:2023-10-30T17:32:25.210+0000 c495640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T17:32:25.588 INFO:tasks.ceph.osd.2.smithi170.stderr:2023-10-30T17:32:25.573+0000 10a5c640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T17:32:25.632 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-30T17:32:25.615+0000 11a5e640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T17:32:25.714 INFO:tasks.ceph.osd.1.smithi170.stderr:2023-10-30T17:32:25.700+0000 10a5c640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T17:32:26.342 INFO:tasks.ceph.osd.0.smithi170.stderr:2023-10-30T17:32:26.330+0000 c495640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T17:32:26.784 INFO:tasks.ceph.osd.0.smithi170.stderr:2023-10-30T17:32:26.770+0000 1125d640 -1 osd.0 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T17:32:30.810 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T17:32:30.907 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.906+0000 7fb01f33e640 10 monclient: get_monmap_and_config 2023-10-30T17:32:30.907 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.906+0000 7fb01f33e640 10 monclient: build_initial_monmap 2023-10-30T17:32:30.907 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.906+0000 7fb01f33e640 10 monclient: monmap: 2023-10-30T17:32:30.908 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:32:30.908 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:30.908 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:32:30.907021+0000 2023-10-30T17:32:30.908 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:32:30.907021+0000 2023-10-30T17:32:30.909 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:30.909 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:30.909 INFO:teuthology.misc.health.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:30.909 INFO:teuthology.misc.health.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:30.910 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:30.910 INFO:teuthology.misc.health.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:30.910 INFO:teuthology.misc.health.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:30.910 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:30.913 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.906+0000 7fb01f33e640 1 Processor -- start 2023-10-30T17:32:30.913 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.906+0000 7fb01f33e640 1 -- start start 2023-10-30T17:32:30.914 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.906+0000 7fb01f33e640 10 monclient: init 2023-10-30T17:32:30.914 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:30.914 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-10-30T17:32:30.914 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0180b44b0 0x7fb0180a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:30.915 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 10 monclient(hunting): picked mon.noname-c con 0x7fb0180b44b0 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:30.915 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb01813ec40 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:30.915 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0181419d0 0x7fb018141e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:30.915 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 10 monclient(hunting): picked mon.noname-b con 0x7fb0181419d0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:30.915 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0180aa4b0 con 0x7fb01813ec40 2023-10-30T17:32:30.915 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:30.916 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180b4890 con 0x7fb0180b44b0 2023-10-30T17:32:30.916 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:30.916 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180aa790 con 0x7fb0181419d0 2023-10-30T17:32:30.916 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:30.916 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.907+0000 7fb01f33e640 10 monclient(hunting): authenticate will time out at 1333.230554s 2023-10-30T17:32:30.916 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01d8db640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0181419d0 0x7fb018141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:30.917 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01c8d9640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fb01813ec40 0x7fb01813f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:51410/0 (socket says 172.21.15.170:51410) 2023-10-30T17:32:30.917 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01c8d9640 1 -- 172.21.15.170:0/889749155 learned_addr learned my addr 172.21.15.170:0/889749155 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:30.917 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01d0da640 1 --2- 172.21.15.170:0/889749155 >> v2:172.21.15.177:3300/0 conn(0x7fb0180b44b0 0x7fb0180a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:30.917 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01d8db640 10 monclient(hunting): get_auth_request con 0x7fb0181419d0 auth_method 0 2023-10-30T17:32:30.917 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01d8db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:30.918 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01d8db640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:30.918 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01d8db640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:30.918 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01d0da640 10 monclient(hunting): get_auth_request con 0x7fb0180b44b0 auth_method 0 2023-10-30T17:32:30.918 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01d0da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:30.919 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01d0da640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:30.919 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.908+0000 7fb01d0da640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:30.919 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.909+0000 7fb01d8db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:30.919 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.909+0000 7fb01d8db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:30.919 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.909+0000 7fb01d8db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:30.920 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.909+0000 7fb01d0da640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:30.920 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.909+0000 7fb01d0da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:30.920 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.909+0000 7fb01d0da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:30.920 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb01d8db640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2023-10-30T17:32:30.921 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb01d8db640 1 -- 172.21.15.170:0/889749155 >> v2:172.21.15.177:3300/0 conn(0x7fb0180b44b0 msgr2=0x7fb0180a8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:30.921 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb00ffff640 1 -- 172.21.15.170:0/889749155 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294751934 0 0) 0x7fb0180aa4b0 con 0x7fb01813ec40 2023-10-30T17:32:30.921 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb01d8db640 1 --2- 172.21.15.170:0/889749155 >> v2:172.21.15.177:3300/0 conn(0x7fb0180b44b0 0x7fb0180a8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:30.921 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb01d8db640 1 -- 172.21.15.170:0/889749155 >> v1:172.21.15.170:6789/0 conn(0x7fb01813ec40 legacy=0x7fb01813f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:30.921 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb01d8db640 10 monclient: _finish_hunting 0 2023-10-30T17:32:30.922 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb01d8db640 1 monclient: found mon.noname-b 2023-10-30T17:32:30.922 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb01d8db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:30.922 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb01d8db640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:30.922 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb01d8db640 1 -- 172.21.15.170:0/889749155 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180a9200 con 0x7fb0181419d0 2023-10-30T17:32:30.922 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.910+0000 7fb00ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T17:32:30.926 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.913+0000 7fb01d8db640 1 --2- 172.21.15.170:0/889749155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0181419d0 0x7fb018141e50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb010009d90 tx=0x7fb01002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39927b40996a607f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:30.926 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 1 -- 172.21.15.170:0/889749155 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01002cce0 con 0x7fb0181419d0 2023-10-30T17:32:30.927 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:30.927 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:30.927 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 10 monclient: dump: 2023-10-30T17:32:30.927 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:30.928 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:30.928 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:30.928 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:30.928 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:30.928 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:30.928 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:30.929 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:30.929 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:30.929 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:30.929 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 10 monclient: _finish_auth 0 2023-10-30T17:32:30.929 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 10 monclient: _check_auth_tickets 2023-10-30T17:32:30.929 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:30.930 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 1 -- 172.21.15.170:0/889749155 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01002ce40 con 0x7fb0181419d0 2023-10-30T17:32:30.930 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb01f33e640 5 monclient: authenticate success, global_id 4220 2023-10-30T17:32:30.930 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:30.930 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb01f33e640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:30.930 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb01f33e640 4 set_mon_vals no callback set 2023-10-30T17:32:30.931 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 1 -- 172.21.15.170:0/889749155 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb010034660 con 0x7fb0181419d0 2023-10-30T17:32:30.931 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:30.931 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:30.931 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.916+0000 7fb00ffff640 10 monclient: dump: 2023-10-30T17:32:30.931 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:30.931 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:30.932 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:30.932 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:30.932 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:30.932 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:30.932 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:30.933 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:30.933 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:30.933 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:30.933 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.917+0000 7fb01f33e640 10 monclient: shutdown 2023-10-30T17:32:30.933 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.917+0000 7fb01f33e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:30.933 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.917+0000 7fb01f33e640 1 -- 172.21.15.170:0/889749155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0181419d0 msgr2=0x7fb018141e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:30.934 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.917+0000 7fb01f33e640 1 --2- 172.21.15.170:0/889749155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0181419d0 0x7fb018141e50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb010009d90 tx=0x7fb01002bf30 comp rx=0 tx=0).stop 2023-10-30T17:32:30.934 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.917+0000 7fb01f33e640 1 -- 172.21.15.170:0/889749155 shutdown_connections 2023-10-30T17:32:30.934 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.917+0000 7fb01f33e640 1 --2- 172.21.15.170:0/889749155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0181419d0 0x7fb018141e50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:30.934 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.917+0000 7fb01f33e640 1 --2- 172.21.15.170:0/889749155 >> v2:172.21.15.177:3300/0 conn(0x7fb0180b44b0 0x7fb0180a8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:30.935 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.917+0000 7fb01f33e640 1 -- 172.21.15.170:0/889749155 >> 172.21.15.170:0/889749155 conn(0x7fb018071b60 msgr2=0x7fb018071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:30.935 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.917+0000 7fb01f33e640 1 -- 172.21.15.170:0/889749155 shutdown_connections 2023-10-30T17:32:30.935 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.917+0000 7fb01f33e640 1 -- 172.21.15.170:0/889749155 wait complete. 2023-10-30T17:32:30.935 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 10 monclient: build_initial_monmap 2023-10-30T17:32:30.935 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 10 monclient: monmap: 2023-10-30T17:32:30.935 INFO:teuthology.misc.health.smithi170.stderr:epoch 0 2023-10-30T17:32:30.936 INFO:teuthology.misc.health.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:30.936 INFO:teuthology.misc.health.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:30.936 INFO:teuthology.misc.health.smithi170.stderr:created 0.000000 2023-10-30T17:32:30.936 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:30.936 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:30.936 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:30.937 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:30.937 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:30.937 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:30.937 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 1 librados: starting msgr at 2023-10-30T17:32:30.937 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 1 librados: starting objecter 2023-10-30T17:32:30.938 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 1 Processor -- start 2023-10-30T17:32:30.938 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 1 -- start start 2023-10-30T17:32:30.938 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 1 librados: setting wanted keys 2023-10-30T17:32:30.938 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 1 librados: calling monclient init 2023-10-30T17:32:30.938 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 10 monclient: init 2023-10-30T17:32:30.939 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:30.939 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:32:30.939 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0180b44b0 0x7fb0181d6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:30.939 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 10 monclient(hunting): picked mon.noname-c con 0x7fb0180b44b0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:30.939 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.918+0000 7fb01f33e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb01813ec40 0x7fb018138ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:30.940 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 10 monclient(hunting): picked mon.noname-a con 0x7fb01813ec40 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:30.940 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0181419d0 0x7fb0180b6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:30.940 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 10 monclient(hunting): picked mon.noname-b con 0x7fb0181419d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:30.940 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:30.940 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0181d7250 con 0x7fb01813ec40 2023-10-30T17:32:30.941 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:30.941 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0181d67f0 con 0x7fb0181419d0 2023-10-30T17:32:30.941 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:30.941 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb018138ff0 con 0x7fb0180b44b0 2023-10-30T17:32:30.941 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:30.941 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01f33e640 10 monclient(hunting): authenticate will time out at 1333.242105s 2023-10-30T17:32:30.942 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01d0da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0180b44b0 0x7fb0181d6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:30.942 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01d8db640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0181419d0 0x7fb0180b6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:30.942 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01c8d9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb01813ec40 0x7fb018138ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:30.942 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01d0da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0180b44b0 0x7fb0181d6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:53096/0 (socket says 172.21.15.170:53096) 2023-10-30T17:32:30.943 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01d0da640 1 -- 172.21.15.170:0/817075316 learned_addr learned my addr 172.21.15.170:0/817075316 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:30.943 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01d0da640 10 monclient(hunting): get_auth_request con 0x7fb0180b44b0 auth_method 0 2023-10-30T17:32:30.943 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01d0da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:30.943 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01d0da640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:30.943 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.919+0000 7fb01d0da640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:30.944 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01c8d9640 10 monclient(hunting): get_auth_request con 0x7fb01813ec40 auth_method 0 2023-10-30T17:32:30.944 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01c8d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:30.944 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01c8d9640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:30.944 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01c8d9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:30.944 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01d8db640 10 monclient(hunting): get_auth_request con 0x7fb0181419d0 auth_method 0 2023-10-30T17:32:30.945 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01d8db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:30.945 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01d8db640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:30.945 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01d8db640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:30.945 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01d0da640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:30.945 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01d0da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:30.945 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01d0da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:30.946 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01d8db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:30.946 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01d8db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:30.946 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01d8db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:30.946 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01c8d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:30.946 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01c8d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:30.947 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.920+0000 7fb01c8d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:30.947 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.923+0000 7fb01d8db640 10 monclient(hunting): handle_auth_done global_id 4255 payload 824 2023-10-30T17:32:30.947 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.923+0000 7fb01d8db640 1 -- 172.21.15.170:0/817075316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0180b44b0 msgr2=0x7fb0181d6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:30.947 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.923+0000 7fb01d8db640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0180b44b0 0x7fb0181d6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:30.947 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.923+0000 7fb01d8db640 1 -- 172.21.15.170:0/817075316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb01813ec40 msgr2=0x7fb018138ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:30.948 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.923+0000 7fb01d8db640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb01813ec40 0x7fb018138ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:30.948 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.923+0000 7fb01d8db640 10 monclient: _finish_hunting 0 2023-10-30T17:32:30.948 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.923+0000 7fb01d8db640 1 monclient: found mon.noname-b 2023-10-30T17:32:30.948 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.923+0000 7fb01d8db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:30.948 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.923+0000 7fb01d8db640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:30.948 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.923+0000 7fb01d8db640 1 -- 172.21.15.170:0/817075316 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb010009a70 con 0x7fb0181419d0 2023-10-30T17:32:30.950 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.926+0000 7fb01d8db640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0181419d0 0x7fb0180b6cc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb010033000 tx=0x7fb010004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:30.950 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.926+0000 7fb00dffb640 10 client.?.objecter ms_handle_connect 0x7fb0181419d0 2023-10-30T17:32:30.950 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.926+0000 7fb00dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:30.951 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 1 -- 172.21.15.170:0/817075316 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01003d600 con 0x7fb0181419d0 2023-10-30T17:32:30.951 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:30.951 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:30.951 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 10 monclient: dump: 2023-10-30T17:32:30.951 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:30.951 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:30.952 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:30.952 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:30.952 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:30.952 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:30.953 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:30.953 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:30.953 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:30.953 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:30.953 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 10 monclient: _finish_auth 0 2023-10-30T17:32:30.954 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 10 monclient: _check_auth_tickets 2023-10-30T17:32:30.954 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:30.954 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 1 -- 172.21.15.170:0/817075316 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01003d780 con 0x7fb0181419d0 2023-10-30T17:32:30.954 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:30.954 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 1 -- 172.21.15.170:0/817075316 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01003db30 con 0x7fb0181419d0 2023-10-30T17:32:30.955 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:30.955 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:30.955 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00dffb640 10 monclient: dump: 2023-10-30T17:32:30.955 INFO:teuthology.misc.health.smithi170.stderr:epoch 1 2023-10-30T17:32:30.955 INFO:teuthology.misc.health.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:30.956 INFO:teuthology.misc.health.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:30.956 INFO:teuthology.misc.health.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:30.956 INFO:teuthology.misc.health.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:30.956 INFO:teuthology.misc.health.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:30.956 INFO:teuthology.misc.health.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:30.956 INFO:teuthology.misc.health.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:30.957 INFO:teuthology.misc.health.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:30.957 INFO:teuthology.misc.health.smithi170.stderr: 2023-10-30T17:32:30.957 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb01f33e640 5 monclient: authenticate success, global_id 4255 2023-10-30T17:32:30.957 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb01f33e640 10 monclient: _renew_subs 2023-10-30T17:32:30.957 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb01f33e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:30.958 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb01f33e640 1 -- 172.21.15.170:0/817075316 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0180b7200 con 0x7fb0181419d0 2023-10-30T17:32:30.958 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb00f7fe640 4 set_mon_vals no callback set 2023-10-30T17:32:30.958 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb01f33e640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:30.958 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb01f33e640 10 monclient: _renew_subs 2023-10-30T17:32:30.958 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb01f33e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:30.958 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb01f33e640 1 -- 172.21.15.170:0/817075316 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0181d6580 con 0x7fb0181419d0 2023-10-30T17:32:30.959 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.931+0000 7fb01f33e640 1 librados: init done 2023-10-30T17:32:30.959 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.932+0000 7fb01f33e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:30.959 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.932+0000 7fb01f33e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:30.959 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.932+0000 7fb01f33e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:30.959 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.932+0000 7fb01f33e640 1 -- 172.21.15.170:0/817075316 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb018067cd0 con 0x7fb0181419d0 2023-10-30T17:32:30.960 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.951+0000 7fb00dffb640 1 -- 172.21.15.170:0/817075316 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fb01003b040 con 0x7fb0181419d0 2023-10-30T17:32:30.961 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.952+0000 7fb00dffb640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7faffc06c7c0 0x7faffc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:30.961 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.952+0000 7fb01d0da640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7faffc06c7c0 0x7faffc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:30.961 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.952+0000 7fb01d0da640 10 monclient: get_auth_request con 0x7faffc06c7c0 auth_method 0 2023-10-30T17:32:30.961 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.952+0000 7fb01d0da640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7faffc06c7c0 0x7faffc06ec80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb014009c00 tx=0x7fb014009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:30.962 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.952+0000 7fb00dffb640 10 client.4255.objecter ms_handle_connect 0x7faffc06c7c0 2023-10-30T17:32:30.962 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.955+0000 7fb00dffb640 1 -- 172.21.15.170:0/817075316 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb01007ab20 con 0x7fb0181419d0 2023-10-30T17:32:30.962 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.955+0000 7fb00dffb640 10 client.4255.objecter ms_dispatch 0x7fb018000b90 osd_map(19..19 src has 1..19) v4 2023-10-30T17:32:30.962 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.955+0000 7fb00dffb640 3 client.4255.objecter handle_osd_map got epochs [19,19] > 0 2023-10-30T17:32:30.962 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.955+0000 7fb00dffb640 3 client.4255.objecter handle_osd_map decoding full epoch 19 2023-10-30T17:32:30.963 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:30.955+0000 7fb00dffb640 20 client.4255.objecter dump_active .. 0 homeless 2023-10-30T17:32:31.196 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.195+0000 7fb00dffb640 1 -- 172.21.15.170:0/817075316 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb010076e60 con 0x7fb0181419d0 2023-10-30T17:32:31.197 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.195+0000 7fb00dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:31.197 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.195+0000 7fb00dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:31.387 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.386+0000 7fb01f33e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:31.387 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.386+0000 7fb01f33e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:31.388 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.386+0000 7fb01f33e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:31.388 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.386+0000 7fb01f33e640 1 -- 172.21.15.170:0/817075316 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb01814c1f0 con 0x7fb0181419d0 2023-10-30T17:32:31.544 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.543+0000 7fb00dffb640 1 -- 172.21.15.170:0/817075316 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12875 (secure 0 0 0) 0x7fb01004d020 con 0x7fb0181419d0 2023-10-30T17:32:31.544 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.543+0000 7fb00dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:31.545 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.543+0000 7fb00dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:31.545 INFO:teuthology.misc.health.smithi170.stdout: 2023-10-30T17:32:31.545 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":19,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:32:30.099733+0000","last_up_change":"2023-10-30T17:32:27.945525+0000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T17:32:25.610766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6809","nonce":500152820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6811","nonce":500152820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6815","nonce":500152820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6813","nonce":500152820}]},"public_addr":"172.21.15.170:6809/500152820","cluster_addr":"172.21.15.170:6811/500152820","heartbeat_back_addr":"172.21.15.170:6815/500152820","heartbeat_front_addr":"172.21.15.170:6813/500152820","state":["exists","up"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6825","nonce":527201623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6827","nonce":527201623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6831","nonce":527201623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6829","nonce":527201623}]},"public_addr":"172.21.15.170:6825/527201623","cluster_addr":"172.21.15.170:6827/527201623","heartbeat_back_addr":"172.21.15.170:6831/527201623","heartbeat_front_addr":"172.21.15.170:6829/527201623","state":["exists","up"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6817","nonce":425342180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6819","nonce":425342180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6823","nonce":425342180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6821","nonce":425342180}]},"public_addr":"172.21.15.170:6817/425342180","cluster_addr":"172.21.15.170:6819/425342180","heartbeat_back_addr":"172.21.15.170:6823/425342180","heartbeat_front_addr":"172.21.15.170:6821/425342180","state":["exists","up"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6801","nonce":3657546070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6803","nonce":3657546070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6807","nonce":3657546070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6805","nonce":3657546070}]},"public_addr":"172.21.15.170:6801/3657546070","cluster_addr":"172.21.15.170:6803/3657546070","heartbeat_back_addr":"172.21.15.170:6807/3657546070","heartbeat_front_addr":"172.21.15.170:6805/3657546070","state":["exists","up"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6803","nonce":2107882913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6805","nonce":2107882913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6809","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6811","nonce":2107882913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6807","nonce":2107882913}]},"public_addr":"172.21.15.177:6803/2107882913","cluster_addr":"172.21.15.177:6805/2107882913","heartbeat_back_addr":"172.21.15.177:6811/2107882913","heartbeat_front_addr":"172.21.15.177:6807/2107882913","state":["exists","up"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6810","nonce":2243649533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6813","nonce":2243649533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6817","nonce":2243649533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6815","nonce":2243649533}]},"public_addr":"172.21.15.177:6810/2243649533","cluster_addr":"172.21.15.177:6813/2243649533","heartbeat_back_addr":"172.21.15.177:6817/2243649533","heartbeat_front_addr":"172.21.15.177:6815/2243649533","state":["exists","up"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6827","nonce":1041798931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6829","nonce":1041798931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6833","nonce":1041798931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6831","nonce":1041798931}]},"public_addr":"172.21.15.177:6827/1041798931","cluster_addr":"172.21.15.177:6829/1041798931","heartbeat_back_addr":"172.21.15.177:6833/1041798931","heartbeat_front_addr":"172.21.15.177:6831/1041798931","state":["exists","up"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6819","nonce":3145923709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6821","nonce":3145923709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6825","nonce":3145923709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6823","nonce":3145923709}]},"public_addr":"172.21.15.177:6819/3145923709","cluster_addr":"172.21.15.177:6821/3145923709","heartbeat_back_addr":"172.21.15.177:6825/3145923709","heartbeat_front_addr":"172.21.15.177:6823/3145923709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:24.231313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:24.066267+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:23.315345+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:23.639773+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.166624+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.275780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.364855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:22.235136+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:32:31.547 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 10 librados: watch_flush enter 2023-10-30T17:32:31.547 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 10 librados: watch_flush exit 2023-10-30T17:32:31.547 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 20 client.4255.objecter shutdown clearing up homeless session... 2023-10-30T17:32:31.547 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 10 client.4255.objecter successfully canceled tick 2023-10-30T17:32:31.548 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 -- 172.21.15.170:0/817075316 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7faffc06c7c0 msgr2=0x7faffc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:31.548 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7faffc06c7c0 0x7faffc06ec80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb014009c00 tx=0x7fb014009190 comp rx=0 tx=0).stop 2023-10-30T17:32:31.548 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 10 monclient: shutdown 2023-10-30T17:32:31.548 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:31.548 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 -- 172.21.15.170:0/817075316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0181419d0 msgr2=0x7fb0180b6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:31.549 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0181419d0 0x7fb0180b6cc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb010033000 tx=0x7fb010004a40 comp rx=0 tx=0).stop 2023-10-30T17:32:31.549 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 -- 172.21.15.170:0/817075316 shutdown_connections 2023-10-30T17:32:31.549 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7faffc06c7c0 0x7faffc06ec80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:31.549 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0181419d0 0x7fb0180b6cc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:31.549 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb01813ec40 0x7fb018138ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:31.550 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 --2- 172.21.15.170:0/817075316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0180b44b0 0x7fb0181d6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:31.550 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 -- 172.21.15.170:0/817075316 >> 172.21.15.170:0/817075316 conn(0x7fb018071b60 msgr2=0x7fb018071f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:31.550 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 -- 172.21.15.170:0/817075316 shutdown_connections 2023-10-30T17:32:31.550 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 -- 172.21.15.170:0/817075316 wait complete. 2023-10-30T17:32:31.550 INFO:teuthology.misc.health.smithi170.stderr:2023-10-30T17:32:31.545+0000 7fb01f33e640 1 librados: shutdown 2023-10-30T17:32:31.556 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-30T17:32:31.557 INFO:tasks.ceph:Creating RBD pool 2023-10-30T17:32:31.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-30T17:32:31.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.662+0000 7f89696cf640 10 monclient: get_monmap_and_config 2023-10-30T17:32:31.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.662+0000 7f89696cf640 10 monclient: build_initial_monmap 2023-10-30T17:32:31.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.662+0000 7f89696cf640 10 monclient: monmap: 2023-10-30T17:32:31.664 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:31.664 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:31.664 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:31.663186+0000 2023-10-30T17:32:31.664 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:31.663186+0000 2023-10-30T17:32:31.665 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:31.665 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:31.665 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:31.665 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:31.665 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:31.666 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:31.666 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:31.666 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:31.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.662+0000 7f89696cf640 1 Processor -- start 2023-10-30T17:32:31.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 1 -- start start 2023-10-30T17:32:31.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 10 monclient: init 2023-10-30T17:32:31.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:31.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T17:32:31.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f8964142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:31.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 10 monclient(hunting): picked mon.noname-b con 0x7f89640b13e0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:31.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89640715d0 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:31.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f896413d100 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:31.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89640710b0 con 0x7f89640715d0 2023-10-30T17:32:31.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8964070d80 con 0x7f896413d100 2023-10-30T17:32:31.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:31.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89640b17c0 con 0x7f89640b13e0 2023-10-30T17:32:31.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:31.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.663+0000 7f89696cf640 10 monclient(hunting): authenticate will time out at 1333.986835s 2023-10-30T17:32:31.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.664+0000 7f89627fc640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f89640715d0 0x7f896413c9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:51420/0 (socket says 172.21.15.170:51420) 2023-10-30T17:32:31.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.664+0000 7f89627fc640 1 -- 172.21.15.170:0/1294320626 learned_addr learned my addr 172.21.15.170:0/1294320626 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:31.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.666+0000 7f8961ffb640 1 -- 172.21.15.170:0/1294320626 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500965617 0 0) 0x7f89640710b0 con 0x7f89640715d0 2023-10-30T17:32:31.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.666+0000 7f8961ffb640 10 monclient(hunting): _negotiate 2023-10-30T17:32:31.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.666+0000 7f8961ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:31.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.666+0000 7f8961ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:31.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.666+0000 7f8961ffb640 10 monclient(hunting): my global_id is 4266 2023-10-30T17:32:31.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.666+0000 7f8961ffb640 1 -- 172.21.15.170:0/1294320626 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f894c003520 con 0x7f89640715d0 2023-10-30T17:32:31.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.667+0000 7f8962ffd640 1 --2- 172.21.15.170:0/1294320626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f8964142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:31.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.668+0000 7f8962ffd640 10 monclient(hunting): get_auth_request con 0x7f89640b13e0 auth_method 0 2023-10-30T17:32:31.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.668+0000 7f8962ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:31.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.668+0000 7f8962ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:31.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.668+0000 7f8962ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:31.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.668+0000 7f8962ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:31.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.668+0000 7f8962ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:31.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.668+0000 7f8962ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:31.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.670+0000 7f8962ffd640 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2023-10-30T17:32:31.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.670+0000 7f8962ffd640 1 -- 172.21.15.170:0/1294320626 >> v1:172.21.15.177:6789/0 conn(0x7f896413d100 legacy=0x7f896414a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:31.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.670+0000 7f8962ffd640 1 -- 172.21.15.170:0/1294320626 >> v1:172.21.15.170:6789/0 conn(0x7f89640715d0 legacy=0x7f896413c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:31.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.670+0000 7f8962ffd640 10 monclient: _finish_hunting 0 2023-10-30T17:32:31.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.670+0000 7f8962ffd640 1 monclient: found mon.noname-b 2023-10-30T17:32:31.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.670+0000 7f8962ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:31.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.670+0000 7f8962ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:31.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.670+0000 7f8962ffd640 1 -- 172.21.15.170:0/1294320626 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8964072400 con 0x7f89640b13e0 2023-10-30T17:32:31.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.672+0000 7f8962ffd640 1 --2- 172.21.15.170:0/1294320626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f8964142ce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f89540097d0 tx=0x7f895402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94eec9f93e6372ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:31.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f8961ffb640 1 -- 172.21.15.170:0/1294320626 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954031890 con 0x7f89640b13e0 2023-10-30T17:32:31.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f8961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:31.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f8961ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:31.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f8961ffb640 10 monclient: dump: 2023-10-30T17:32:31.680 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:31.680 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:31.680 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:31.680 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:31.680 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:31.681 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:31.681 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:31.681 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:31.681 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:31.681 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:31.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f8961ffb640 10 monclient: _finish_auth 0 2023-10-30T17:32:31.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f8961ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:32:31.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f8961ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:31.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f8961ffb640 1 -- 172.21.15.170:0/1294320626 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89540319f0 con 0x7f89640b13e0 2023-10-30T17:32:31.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f89696cf640 5 monclient: authenticate success, global_id 4226 2023-10-30T17:32:31.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f89696cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.677+0000 7f8961ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f8961ffb640 1 -- 172.21.15.170:0/1294320626 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954031d80 con 0x7f89640b13e0 2023-10-30T17:32:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 4 set_mon_vals no callback set 2023-10-30T17:32:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f8961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f8961ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:31.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f8961ffb640 10 monclient: dump: 2023-10-30T17:32:31.684 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:31.684 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:31.684 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:31.684 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:31.685 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:31.685 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:31.685 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:31.685 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:31.685 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:31.686 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:31.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 10 monclient: shutdown 2023-10-30T17:32:31.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:31.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 1 -- 172.21.15.170:0/1294320626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 msgr2=0x7f8964142ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:31.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 1 --2- 172.21.15.170:0/1294320626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f8964142ce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f89540097d0 tx=0x7f895402b930 comp rx=0 tx=0).stop 2023-10-30T17:32:31.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 1 -- 172.21.15.170:0/1294320626 shutdown_connections 2023-10-30T17:32:31.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 1 --2- 172.21.15.170:0/1294320626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f8964142ce0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:31.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 1 -- 172.21.15.170:0/1294320626 >> 172.21.15.170:0/1294320626 conn(0x7f89640b4610 msgr2=0x7f89640b0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:31.690 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 1 -- 172.21.15.170:0/1294320626 shutdown_connections 2023-10-30T17:32:31.690 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.678+0000 7f89696cf640 1 -- 172.21.15.170:0/1294320626 wait complete. 2023-10-30T17:32:31.690 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.679+0000 7f89696cf640 10 monclient: build_initial_monmap 2023-10-30T17:32:31.690 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.679+0000 7f89696cf640 10 monclient: monmap: 2023-10-30T17:32:31.690 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:31.690 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:31.691 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:31.691 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:31.691 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:31.691 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:31.691 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:31.692 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:31.692 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:31.692 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:31.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.679+0000 7f89696cf640 1 librados: starting msgr at 2023-10-30T17:32:31.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.679+0000 7f89696cf640 1 librados: starting objecter 2023-10-30T17:32:31.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 1 Processor -- start 2023-10-30T17:32:31.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 1 -- start start 2023-10-30T17:32:31.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 1 librados: setting wanted keys 2023-10-30T17:32:31.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 1 librados: calling monclient init 2023-10-30T17:32:31.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient: init 2023-10-30T17:32:31.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:31.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:32:31.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89640715d0 0x7f89640aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:31.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient(hunting): picked mon.noname-a con 0x7f89640715d0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:31.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f89640ae940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:31.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient(hunting): picked mon.noname-c con 0x7f89640b13e0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:31.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f896413d100 0x7f89640aee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:31.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient(hunting): picked mon.noname-b con 0x7f896413d100 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:31.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:31.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89640aa960 con 0x7f89640715d0 2023-10-30T17:32:31.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:31.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89640b0630 con 0x7f896413d100 2023-10-30T17:32:31.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:31.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89640b07a0 con 0x7f89640b13e0 2023-10-30T17:32:31.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:31.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.680+0000 7f89696cf640 10 monclient(hunting): authenticate will time out at 1334.004033s 2023-10-30T17:32:31.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.682+0000 7f8962ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89640715d0 0x7f89640aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:31.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.682+0000 7f89637fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f896413d100 0x7f89640aee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.686+0000 7f89627fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f89640ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.686+0000 7f89627fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f89640ae940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:53118/0 (socket says 172.21.15.170:53118) 2023-10-30T17:32:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.686+0000 7f89627fc640 1 -- 172.21.15.170:0/2836285942 learned_addr learned my addr 172.21.15.170:0/2836285942 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.686+0000 7f89627fc640 10 monclient(hunting): get_auth_request con 0x7f89640b13e0 auth_method 0 2023-10-30T17:32:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.686+0000 7f89627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:31.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.686+0000 7f89627fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:31.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.686+0000 7f89627fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:31.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.687+0000 7f89627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:31.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.687+0000 7f89627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:31.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.687+0000 7f89627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:31.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.690+0000 7f89627fc640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2023-10-30T17:32:31.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.690+0000 7f89627fc640 1 -- 172.21.15.170:0/2836285942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f896413d100 msgr2=0x7f89640aee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:31.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.690+0000 7f89627fc640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f896413d100 0x7f89640aee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:31.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.690+0000 7f89627fc640 1 -- 172.21.15.170:0/2836285942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89640715d0 msgr2=0x7f89640aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:31.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.690+0000 7f89627fc640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89640715d0 0x7f89640aa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:31.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.690+0000 7f89627fc640 10 monclient: _finish_hunting 0 2023-10-30T17:32:31.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.690+0000 7f89627fc640 1 monclient: found mon.noname-c 2023-10-30T17:32:31.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.690+0000 7f89627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:31.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.690+0000 7f89627fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:31.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.690+0000 7f89627fc640 1 -- 172.21.15.170:0/2836285942 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8954009400 con 0x7f89640b13e0 2023-10-30T17:32:31.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.738+0000 7f89627fc640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f89640ae940 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f895800c500 tx=0x7f895800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:31.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.738+0000 7f894b7fe640 10 client.?.objecter ms_handle_connect 0x7f89640b13e0 2023-10-30T17:32:31.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.738+0000 7f894b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:31.745 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 1 -- 172.21.15.170:0/2836285942 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958005ce0 con 0x7f89640b13e0 2023-10-30T17:32:31.745 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:31.745 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:31.745 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 10 monclient: dump: 2023-10-30T17:32:31.745 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:31.746 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:31.746 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:31.746 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:31.746 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:31.746 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:31.747 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:31.747 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:31.747 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:31.747 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:31.747 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 10 monclient: _finish_auth 0 2023-10-30T17:32:31.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:32:31.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:31.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 1 -- 172.21.15.170:0/2836285942 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8958005e40 con 0x7f89640b13e0 2023-10-30T17:32:31.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:31.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 1 -- 172.21.15.170:0/2836285942 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958007330 con 0x7f89640b13e0 2023-10-30T17:32:31.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:31.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:31.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f894b7fe640 10 monclient: dump: 2023-10-30T17:32:31.749 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:31.749 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:31.749 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:31.750 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:31.750 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:31.750 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:31.750 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:31.750 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:31.751 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:31.751 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:31.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f89696cf640 5 monclient: authenticate success, global_id 4229 2023-10-30T17:32:31.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f89696cf640 10 monclient: _renew_subs 2023-10-30T17:32:31.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f89696cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:31.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f89617fa640 4 set_mon_vals no callback set 2023-10-30T17:32:31.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f89696cf640 1 -- 172.21.15.170:0/2836285942 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89640af3c0 con 0x7f89640b13e0 2023-10-30T17:32:31.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f89696cf640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:31.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f89696cf640 10 monclient: _renew_subs 2023-10-30T17:32:31.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f89696cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:31.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f89696cf640 1 -- 172.21.15.170:0/2836285942 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89640a9710 con 0x7f89640b13e0 2023-10-30T17:32:31.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.743+0000 7f89696cf640 1 librados: init done 2023-10-30T17:32:31.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.744+0000 7f89696cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:31.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.744+0000 7f89696cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:31.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.744+0000 7f89696cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:31.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.744+0000 7f89696cf640 1 -- 172.21.15.170:0/2836285942 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89640aa710 con 0x7f89640b13e0 2023-10-30T17:32:31.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.763+0000 7f894b7fe640 1 -- 172.21.15.170:0/2836285942 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f8958007490 con 0x7f89640b13e0 2023-10-30T17:32:31.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.764+0000 7f894b7fe640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f895006c7c0 0x7f895006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:31.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.764+0000 7f8962ffd640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f895006c7c0 0x7f895006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:31.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.764+0000 7f8962ffd640 10 monclient: get_auth_request con 0x7f895006c7c0 auth_method 0 2023-10-30T17:32:31.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.765+0000 7f894b7fe640 10 client.4229.objecter ms_handle_connect 0x7f895006c7c0 2023-10-30T17:32:31.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.765+0000 7f8962ffd640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f895006c7c0 0x7f895006ec80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8954036000 tx=0x7f8954008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:31.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.770+0000 7f894b7fe640 1 -- 172.21.15.170:0/2836285942 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8958007740 con 0x7f89640b13e0 2023-10-30T17:32:31.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.770+0000 7f894b7fe640 10 client.4229.objecter ms_dispatch 0x7f8964000b90 osd_map(19..19 src has 1..19) v4 2023-10-30T17:32:31.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.770+0000 7f894b7fe640 3 client.4229.objecter handle_osd_map got epochs [19,19] > 0 2023-10-30T17:32:31.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.770+0000 7f894b7fe640 3 client.4229.objecter handle_osd_map decoding full epoch 19 2023-10-30T17:32:31.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:31.770+0000 7f894b7fe640 20 client.4229.objecter dump_active .. 0 homeless 2023-10-30T17:32:32.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:32.024+0000 7f894b7fe640 1 -- 172.21.15.170:0/2836285942 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8958057d10 con 0x7f89640b13e0 2023-10-30T17:32:32.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:32.024+0000 7f894b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:32.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:32.024+0000 7f894b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:32.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:32.212+0000 7f89696cf640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T17:32:32.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:32.212+0000 7f89696cf640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T17:32:32.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:32.212+0000 7f89696cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:32.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:32.212+0000 7f89696cf640 1 -- 172.21.15.170:0/2836285942 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f89640aaad0 con 0x7f89640b13e0 2023-10-30T17:32:33.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.218+0000 7f894b7fe640 1 -- 172.21.15.170:0/2836285942 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8958054650 con 0x7f89640b13e0 2023-10-30T17:32:33.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.219+0000 7f894b7fe640 1 -- 172.21.15.170:0/2836285942 <== mon.2 v2:172.21.15.170:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v20) v1 ==== 109+0+0 (secure 0 0 0) 0x7f895808a5e0 con 0x7f89640b13e0 2023-10-30T17:32:33.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.219+0000 7f894b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T17:32:33.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.219+0000 7f894b7fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-10-30T17:32:33.221 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'rbd' created 2023-10-30T17:32:33.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 10 librados: watch_flush enter 2023-10-30T17:32:33.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 10 librados: watch_flush exit 2023-10-30T17:32:33.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 20 client.4229.objecter shutdown clearing up homeless session... 2023-10-30T17:32:33.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 10 client.4229.objecter successfully canceled tick 2023-10-30T17:32:33.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 -- 172.21.15.170:0/2836285942 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f895006c7c0 msgr2=0x7f895006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:33.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f895006c7c0 0x7f895006ec80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8954036000 tx=0x7f8954008030 comp rx=0 tx=0).stop 2023-10-30T17:32:33.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 10 monclient: shutdown 2023-10-30T17:32:33.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:33.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 -- 172.21.15.170:0/2836285942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 msgr2=0x7f89640ae940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:33.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f89640ae940 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f895800c500 tx=0x7f895800c9c0 comp rx=0 tx=0).stop 2023-10-30T17:32:33.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 -- 172.21.15.170:0/2836285942 shutdown_connections 2023-10-30T17:32:33.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f895006c7c0 0x7f895006ec80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:33.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f896413d100 0x7f89640aee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:33.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89640b13e0 0x7f89640ae940 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:33.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 --2- 172.21.15.170:0/2836285942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89640715d0 0x7f89640aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:33.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 -- 172.21.15.170:0/2836285942 >> 172.21.15.170:0/2836285942 conn(0x7f89640b4610 msgr2=0x7f89640b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:33.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 -- 172.21.15.170:0/2836285942 shutdown_connections 2023-10-30T17:32:33.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 -- 172.21.15.170:0/2836285942 wait complete. 2023-10-30T17:32:33.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.221+0000 7f89696cf640 1 librados: shutdown 2023-10-30T17:32:33.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-30T17:32:33.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.336+0000 7f4f8cdc9640 10 monclient: get_monmap_and_config 2023-10-30T17:32:33.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.336+0000 7f4f8cdc9640 10 monclient: build_initial_monmap 2023-10-30T17:32:33.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.336+0000 7f4f8cdc9640 10 monclient: monmap: 2023-10-30T17:32:33.338 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:33.338 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:33.338 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:33.337015+0000 2023-10-30T17:32:33.338 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:33.337015+0000 2023-10-30T17:32:33.339 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:33.339 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:33.339 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:33.339 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:33.339 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:33.339 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:33.340 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:33.340 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:33.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.336+0000 7f4f8cdc9640 1 Processor -- start 2023-10-30T17:32:33.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.336+0000 7f4f8cdc9640 1 -- start start 2023-10-30T17:32:33.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.336+0000 7f4f8cdc9640 10 monclient: init 2023-10-30T17:32:33.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:33.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-10-30T17:32:33.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f880b44c0 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:33.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4f8813eb50 0x7f4f8813efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:33.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8813eb50 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:33.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f880ae7a0 0x7f4f880aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:33.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 10 monclient(hunting): picked mon.noname-a con 0x7f4f880ae7a0 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:33.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f880b0170 con 0x7f4f880b44c0 2023-10-30T17:32:33.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:33.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f880b1770 con 0x7f4f880ae7a0 2023-10-30T17:32:33.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:33.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f880b48a0 con 0x7f4f8813eb50 2023-10-30T17:32:33.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:33.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.337+0000 7f4f8cdc9640 10 monclient(hunting): authenticate will time out at 1335.661017s 2023-10-30T17:32:33.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.338+0000 7f4f85d9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4f8813eb50 0x7f4f8813efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:33.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.338+0000 7f4f86d9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f880ae7a0 0x7f4f880aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:33.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.338+0000 7f4f85d9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4f8813eb50 0x7f4f8813efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60382/0 (socket says 172.21.15.170:60382) 2023-10-30T17:32:33.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.338+0000 7f4f85d9b640 1 -- 172.21.15.170:0/1086212380 learned_addr learned my addr 172.21.15.170:0/1086212380 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:33.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.338+0000 7f4f85d9b640 10 monclient(hunting): get_auth_request con 0x7f4f8813eb50 auth_method 0 2023-10-30T17:32:33.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.338+0000 7f4f85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:33.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.338+0000 7f4f85d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:33.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.338+0000 7f4f85d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:33.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.339+0000 7f4f86d9d640 10 monclient(hunting): get_auth_request con 0x7f4f880ae7a0 auth_method 0 2023-10-30T17:32:33.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.339+0000 7f4f86d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:33.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.339+0000 7f4f86d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:33.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.339+0000 7f4f86d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:33.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.339+0000 7f4f85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:33.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.339+0000 7f4f85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:33.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.339+0000 7f4f85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:33.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.339+0000 7f4f86d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:33.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.339+0000 7f4f86d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:33.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.339+0000 7f4f86d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:33.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.341+0000 7f4f86d9d640 10 monclient(hunting): handle_auth_done global_id 4269 payload 274 2023-10-30T17:32:33.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.341+0000 7f4f86d9d640 1 -- 172.21.15.170:0/1086212380 >> v2:172.21.15.177:3300/0 conn(0x7f4f8813eb50 msgr2=0x7f4f8813efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:33.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.341+0000 7f4f86d9d640 1 --2- 172.21.15.170:0/1086212380 >> v2:172.21.15.177:3300/0 conn(0x7f4f8813eb50 0x7f4f8813efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:33.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.341+0000 7f4f86d9d640 1 -- 172.21.15.170:0/1086212380 >> v1:172.21.15.177:6789/0 conn(0x7f4f880b44c0 legacy=0x7f4f880b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:33.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.341+0000 7f4f86d9d640 10 monclient: _finish_hunting 0 2023-10-30T17:32:33.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.341+0000 7f4f86d9d640 1 monclient: found mon.noname-a 2023-10-30T17:32:33.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.341+0000 7f4f86d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:33.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.341+0000 7f4f86d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:33.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.341+0000 7f4f86d9d640 1 -- 172.21.15.170:0/1086212380 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f880b12e0 con 0x7f4f880ae7a0 2023-10-30T17:32:33.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.343+0000 7f4f86d9d640 1 --2- 172.21.15.170:0/1086212380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f880ae7a0 0x7f4f880aec30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f880b3690 tx=0x7f4f8002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbaaf6463b75182a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:33.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 1 -- 172.21.15.170:0/1086212380 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8002cbf0 con 0x7f4f880ae7a0 2023-10-30T17:32:33.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:33.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:33.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 10 monclient: dump: 2023-10-30T17:32:33.358 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:33.359 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:33.359 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:33.359 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:33.359 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:33.359 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:33.360 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:33.360 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:33.360 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:33.360 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:33.360 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 10 monclient: _finish_auth 0 2023-10-30T17:32:33.360 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 10 monclient: _check_auth_tickets 2023-10-30T17:32:33.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:33.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 1 -- 172.21.15.170:0/1086212380 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8002cd50 con 0x7f4f880ae7a0 2023-10-30T17:32:33.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8cdc9640 5 monclient: authenticate success, global_id 4269 2023-10-30T17:32:33.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8cdc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:33.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:33.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8cdc9640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:33.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 1 -- 172.21.15.170:0/1086212380 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80034660 con 0x7f4f880ae7a0 2023-10-30T17:32:33.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8cdc9640 4 set_mon_vals no callback set 2023-10-30T17:32:33.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:33.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:33.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8559a640 10 monclient: dump: 2023-10-30T17:32:33.363 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:33.363 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:33.363 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:33.363 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:33.363 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:33.364 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:33.364 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:33.364 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:33.364 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:33.364 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:33.364 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.347+0000 7f4f8cdc9640 10 monclient: shutdown 2023-10-30T17:32:33.365 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.348+0000 7f4f8cdc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:33.365 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.348+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/1086212380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f880ae7a0 msgr2=0x7f4f880aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:33.365 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.348+0000 7f4f8cdc9640 1 --2- 172.21.15.170:0/1086212380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f880ae7a0 0x7f4f880aec30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f880b3690 tx=0x7f4f8002be40 comp rx=0 tx=0).stop 2023-10-30T17:32:33.365 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.348+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/1086212380 shutdown_connections 2023-10-30T17:32:33.365 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.348+0000 7f4f8cdc9640 1 --2- 172.21.15.170:0/1086212380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f880ae7a0 0x7f4f880aec30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:33.366 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.348+0000 7f4f8cdc9640 1 --2- 172.21.15.170:0/1086212380 >> v2:172.21.15.177:3300/0 conn(0x7f4f8813eb50 0x7f4f8813efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:33.366 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.348+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/1086212380 >> 172.21.15.170:0/1086212380 conn(0x7f4f88071b60 msgr2=0x7f4f88071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:33.366 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.348+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/1086212380 shutdown_connections 2023-10-30T17:32:33.366 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.348+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/1086212380 wait complete. 2023-10-30T17:32:33.366 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.349+0000 7f4f8cdc9640 10 monclient: build_initial_monmap 2023-10-30T17:32:33.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.349+0000 7f4f8cdc9640 10 monclient: monmap: 2023-10-30T17:32:33.367 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:33.367 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:33.367 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:33.367 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:33.368 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:33.368 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:33.368 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:33.368 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:33.368 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:33.369 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:33.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.349+0000 7f4f8cdc9640 1 librados: starting msgr at 2023-10-30T17:32:33.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.349+0000 7f4f8cdc9640 1 librados: starting objecter 2023-10-30T17:32:33.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.349+0000 7f4f8cdc9640 1 Processor -- start 2023-10-30T17:32:33.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.349+0000 7f4f8cdc9640 1 -- start start 2023-10-30T17:32:33.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.349+0000 7f4f8cdc9640 1 librados: setting wanted keys 2023-10-30T17:32:33.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.349+0000 7f4f8cdc9640 1 librados: calling monclient init 2023-10-30T17:32:33.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.349+0000 7f4f8cdc9640 10 monclient: init 2023-10-30T17:32:33.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:33.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:32:33.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f880ae7a0 0x7f4f88142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:33.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 10 monclient(hunting): picked mon.noname-b con 0x7f4f880ae7a0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:33.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f880b44c0 0x7f4f88142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:33.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 10 monclient(hunting): picked mon.noname-a con 0x7f4f880b44c0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:33.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f8813eb50 0x7f4f881434b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:33.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8813eb50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:33.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:33.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88142a60 con 0x7f4f880b44c0 2023-10-30T17:32:33.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:33.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88145c40 con 0x7f4f880ae7a0 2023-10-30T17:32:33.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:33.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88144cb0 con 0x7f4f8813eb50 2023-10-30T17:32:33.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:33.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.350+0000 7f4f8cdc9640 10 monclient(hunting): authenticate will time out at 1335.673897s 2023-10-30T17:32:33.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f8659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f880ae7a0 0x7f4f88142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:33.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f86d9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f8813eb50 0x7f4f881434b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:33.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f8659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f880ae7a0 0x7f4f88142160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60384/0 (socket says 172.21.15.170:60384) 2023-10-30T17:32:33.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f8659c640 1 -- 172.21.15.170:0/3830379312 learned_addr learned my addr 172.21.15.170:0/3830379312 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:33.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f8659c640 10 monclient(hunting): get_auth_request con 0x7f4f880ae7a0 auth_method 0 2023-10-30T17:32:33.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:33.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f8659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:33.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f8659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:33.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f86d9d640 10 monclient(hunting): get_auth_request con 0x7f4f8813eb50 auth_method 0 2023-10-30T17:32:33.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f86d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:33.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f86d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:33.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.351+0000 7f4f86d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:33.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f86d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:33.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f86d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:33.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f85d9b640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f880b44c0 0x7f4f88142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:33.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f86d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:33.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:33.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:33.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:33.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f85d9b640 10 monclient(hunting): get_auth_request con 0x7f4f880b44c0 auth_method 0 2023-10-30T17:32:33.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:33.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f85d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:33.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.352+0000 7f4f85d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:33.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.353+0000 7f4f85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:33.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.353+0000 7f4f85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:33.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.353+0000 7f4f85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:33.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.356+0000 7f4f8659c640 10 monclient(hunting): handle_auth_done global_id 4261 payload 824 2023-10-30T17:32:33.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.356+0000 7f4f8659c640 1 -- 172.21.15.170:0/3830379312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f8813eb50 msgr2=0x7f4f881434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:33.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.356+0000 7f4f8659c640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f8813eb50 0x7f4f881434b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:33.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.356+0000 7f4f8659c640 1 -- 172.21.15.170:0/3830379312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f880b44c0 msgr2=0x7f4f88142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:33.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.356+0000 7f4f8659c640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f880b44c0 0x7f4f88142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:33.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.356+0000 7f4f8659c640 10 monclient: _finish_hunting 0 2023-10-30T17:32:33.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.356+0000 7f4f8659c640 1 monclient: found mon.noname-b 2023-10-30T17:32:33.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.356+0000 7f4f8659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:33.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.356+0000 7f4f8659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:33.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.356+0000 7f4f8659c640 1 -- 172.21.15.170:0/3830379312 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f80009a70 con 0x7f4f880ae7a0 2023-10-30T17:32:33.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.359+0000 7f4f8659c640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f880ae7a0 0x7f4f88142160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800c480 tx=0x7f4f7800c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:33.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.359+0000 7f4f76ffd640 10 client.?.objecter ms_handle_connect 0x7f4f880ae7a0 2023-10-30T17:32:33.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.359+0000 7f4f76ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:33.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 1 -- 172.21.15.170:0/3830379312 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f780052d0 con 0x7f4f880ae7a0 2023-10-30T17:32:33.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:33.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:33.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 10 monclient: dump: 2023-10-30T17:32:33.383 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:33.383 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:33.383 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:33.383 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:33.384 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:33.384 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:33.384 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:33.384 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:33.384 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:33.385 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:33.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 10 monclient: _finish_auth 0 2023-10-30T17:32:33.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:33.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:33.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 1 -- 172.21.15.170:0/3830379312 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f78005430 con 0x7f4f880ae7a0 2023-10-30T17:32:33.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:33.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 1 -- 172.21.15.170:0/3830379312 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78012870 con 0x7f4f880ae7a0 2023-10-30T17:32:33.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f8cdc9640 5 monclient: authenticate success, global_id 4261 2023-10-30T17:32:33.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f8cdc9640 10 monclient: _renew_subs 2023-10-30T17:32:33.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f8cdc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:33.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/3830379312 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f881439f0 con 0x7f4f880ae7a0 2023-10-30T17:32:33.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f84d99640 4 set_mon_vals no callback set 2023-10-30T17:32:33.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:33.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:33.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f76ffd640 10 monclient: dump: 2023-10-30T17:32:33.388 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:33.388 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:33.388 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:33.388 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:33.388 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:33.388 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:33.389 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:33.389 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:33.389 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:33.389 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:33.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f8cdc9640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:33.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f8cdc9640 10 monclient: _renew_subs 2023-10-30T17:32:33.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f8cdc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:33.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/3830379312 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f881362c0 con 0x7f4f880ae7a0 2023-10-30T17:32:33.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.362+0000 7f4f8cdc9640 1 librados: init done 2023-10-30T17:32:33.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.363+0000 7f4f8cdc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:33.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.363+0000 7f4f8cdc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:33.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.363+0000 7f4f8cdc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:33.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.363+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/3830379312 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f88145ab0 con 0x7f4f880ae7a0 2023-10-30T17:32:33.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.382+0000 7f4f76ffd640 1 -- 172.21.15.170:0/3830379312 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4f780129d0 con 0x7f4f880ae7a0 2023-10-30T17:32:33.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.382+0000 7f4f76ffd640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4f6806c800 0x7f4f6806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:33.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.382+0000 7f4f85d9b640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4f6806c800 0x7f4f6806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:33.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.382+0000 7f4f85d9b640 10 monclient: get_auth_request con 0x7f4f6806c800 auth_method 0 2023-10-30T17:32:33.393 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.383+0000 7f4f85d9b640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4f6806c800 0x7f4f6806ecc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4f88141620 tx=0x7f4f7c009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:33.393 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.383+0000 7f4f76ffd640 10 client.4261.objecter ms_handle_connect 0x7f4f6806c800 2023-10-30T17:32:33.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.620+0000 7f4f76ffd640 1 -- 172.21.15.170:0/3830379312 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f4f78058990 con 0x7f4f880ae7a0 2023-10-30T17:32:33.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.620+0000 7f4f76ffd640 10 client.4261.objecter ms_dispatch 0x7f4f88000b90 osd_map(20..20 src has 1..20) v4 2023-10-30T17:32:33.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.620+0000 7f4f76ffd640 3 client.4261.objecter handle_osd_map got epochs [20,20] > 0 2023-10-30T17:32:33.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.620+0000 7f4f76ffd640 3 client.4261.objecter handle_osd_map decoding full epoch 20 2023-10-30T17:32:33.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.620+0000 7f4f76ffd640 20 client.4261.objecter dump_active .. 0 homeless 2023-10-30T17:32:33.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.638+0000 7f4f76ffd640 1 -- 172.21.15.170:0/3830379312 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4f78054cd0 con 0x7f4f880ae7a0 2023-10-30T17:32:33.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.638+0000 7f4f76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:33.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.638+0000 7f4f76ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:33.818 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.817+0000 7f4f8cdc9640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T17:32:33.818 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.817+0000 7f4f8cdc9640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T17:32:33.819 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.817+0000 7f4f8cdc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:33.819 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:33.817+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/3830379312 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4f88067cd0 con 0x7f4f880ae7a0 2023-10-30T17:32:35.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.004+0000 7f4f76ffd640 1 -- 172.21.15.170:0/3830379312 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v22) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4f780582f0 con 0x7f4f880ae7a0 2023-10-30T17:32:35.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.004+0000 7f4f76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T17:32:35.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.004+0000 7f4f76ffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-10-30T17:32:35.006 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-30T17:32:35.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 10 librados: watch_flush enter 2023-10-30T17:32:35.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 10 librados: watch_flush exit 2023-10-30T17:32:35.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 20 client.4261.objecter shutdown clearing up homeless session... 2023-10-30T17:32:35.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 10 client.4261.objecter successfully canceled tick 2023-10-30T17:32:35.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/3830379312 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4f6806c800 msgr2=0x7f4f6806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4f6806c800 0x7f4f6806ecc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4f88141620 tx=0x7f4f7c009240 comp rx=0 tx=0).stop 2023-10-30T17:32:35.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 10 monclient: shutdown 2023-10-30T17:32:35.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:35.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/3830379312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f880ae7a0 msgr2=0x7f4f88142160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f880ae7a0 0x7f4f88142160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800c480 tx=0x7f4f7800c940 comp rx=0 tx=0).stop 2023-10-30T17:32:35.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/3830379312 shutdown_connections 2023-10-30T17:32:35.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4f6806c800 0x7f4f6806ecc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f8813eb50 0x7f4f881434b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f880b44c0 0x7f4f88142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 --2- 172.21.15.170:0/3830379312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f880ae7a0 0x7f4f88142160 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/3830379312 >> 172.21.15.170:0/3830379312 conn(0x7f4f88071b60 msgr2=0x7f4f8813bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:35.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/3830379312 shutdown_connections 2023-10-30T17:32:35.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 -- 172.21.15.170:0/3830379312 wait complete. 2023-10-30T17:32:35.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.007+0000 7f4f8cdc9640 1 librados: shutdown 2023-10-30T17:32:35.020 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T17:32:35.021 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T17:32:35.021 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T17:32:35.022 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T17:32:35.023 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T17:32:35.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.125+0000 7f35e807c640 10 monclient: get_monmap_and_config 2023-10-30T17:32:35.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.125+0000 7f35e807c640 10 monclient: build_initial_monmap 2023-10-30T17:32:35.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.125+0000 7f35e807c640 10 monclient: monmap: 2023-10-30T17:32:35.127 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:35.127 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:35.127 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:35.126110+0000 2023-10-30T17:32:35.127 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:35.126110+0000 2023-10-30T17:32:35.128 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:35.128 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.128 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:35.128 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:35.128 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:35.129 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:35.129 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:35.129 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.125+0000 7f35e807c640 1 Processor -- start 2023-10-30T17:32:35.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.125+0000 7f35e807c640 1 -- start start 2023-10-30T17:32:35.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.125+0000 7f35e807c640 10 monclient: init 2023-10-30T17:32:35.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:35.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-10-30T17:32:35.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35e00b7c90 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:35.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35e013eb60 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:35.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e0144f50 0x7f35e0145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 10 monclient(hunting): picked mon.noname-b con 0x7f35e0144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:35.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00aa740 con 0x7f35e013eb60 2023-10-30T17:32:35.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00aa9b0 con 0x7f35e00b7c90 2023-10-30T17:32:35.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:35.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00a9500 con 0x7f35e0144f50 2023-10-30T17:32:35.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:35.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.126+0000 7f35e807c640 10 monclient(hunting): authenticate will time out at 1337.449598s 2023-10-30T17:32:35.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.127+0000 7f35e6619640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e0144f50 0x7f35e0145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.127+0000 7f35e5617640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f35e013eb60 0x7f35e013efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:51434/0 (socket says 172.21.15.170:51434) 2023-10-30T17:32:35.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.127+0000 7f35e5617640 1 -- 172.21.15.170:0/226655779 learned_addr learned my addr 172.21.15.170:0/226655779 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:35.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.127+0000 7f35e6619640 10 monclient(hunting): get_auth_request con 0x7f35e0144f50 auth_method 0 2023-10-30T17:32:35.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.127+0000 7f35e6619640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:35.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.127+0000 7f35e6619640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.127+0000 7f35e6619640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.128+0000 7f35e6619640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:35.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.128+0000 7f35e6619640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:35.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.128+0000 7f35e6619640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:35.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.129+0000 7f35e4e16640 1 -- 172.21.15.170:0/226655779 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455624850 0 0) 0x7f35e00aa740 con 0x7f35e013eb60 2023-10-30T17:32:35.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.129+0000 7f35e4e16640 10 monclient(hunting): _negotiate 2023-10-30T17:32:35.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.129+0000 7f35e4e16640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.129+0000 7f35e4e16640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.129+0000 7f35e4e16640 10 monclient(hunting): my global_id is 4275 2023-10-30T17:32:35.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.129+0000 7f35e4e16640 1 -- 172.21.15.170:0/226655779 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35cc003520 con 0x7f35e013eb60 2023-10-30T17:32:35.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.130+0000 7f35e6619640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2023-10-30T17:32:35.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.130+0000 7f35e6619640 1 -- 172.21.15.170:0/226655779 >> v1:172.21.15.177:6789/0 conn(0x7f35e00b7c90 legacy=0x7f35e00a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.130+0000 7f35e6619640 1 -- 172.21.15.170:0/226655779 >> v1:172.21.15.170:6789/0 conn(0x7f35e013eb60 legacy=0x7f35e013efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.130+0000 7f35e6619640 10 monclient: _finish_hunting 0 2023-10-30T17:32:35.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.130+0000 7f35e6619640 1 monclient: found mon.noname-b 2023-10-30T17:32:35.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.130+0000 7f35e6619640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:35.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.130+0000 7f35e6619640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:35.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.130+0000 7f35e6619640 1 -- 172.21.15.170:0/226655779 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00a9900 con 0x7f35e0144f50 2023-10-30T17:32:35.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.132+0000 7f35e6619640 1 --2- 172.21.15.170:0/226655779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e0144f50 0x7f35e0145400 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0097d0 tx=0x7f35dc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c072a6b54c762536 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:35.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 1 -- 172.21.15.170:0/226655779 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc031800 con 0x7f35e0144f50 2023-10-30T17:32:35.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:35.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:35.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 10 monclient: dump: 2023-10-30T17:32:35.144 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:35.144 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:35.144 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.144 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.145 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:35.145 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.145 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:35.145 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:35.146 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:35.146 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 10 monclient: _finish_auth 0 2023-10-30T17:32:35.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 10 monclient: _check_auth_tickets 2023-10-30T17:32:35.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:35.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 1 -- 172.21.15.170:0/226655779 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc031960 con 0x7f35e0144f50 2023-10-30T17:32:35.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:35.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 1 -- 172.21.15.170:0/226655779 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc031cf0 con 0x7f35e0144f50 2023-10-30T17:32:35.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e807c640 5 monclient: authenticate success, global_id 4235 2023-10-30T17:32:35.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e807c640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:35.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:35.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e807c640 4 set_mon_vals no callback set 2023-10-30T17:32:35.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:35.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e4e16640 10 monclient: dump: 2023-10-30T17:32:35.148 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:35.149 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:35.149 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.149 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.149 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:35.150 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.150 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:35.150 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:35.150 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:35.150 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e807c640 10 monclient: shutdown 2023-10-30T17:32:35.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e807c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:35.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e807c640 1 -- 172.21.15.170:0/226655779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e0144f50 msgr2=0x7f35e0145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.136+0000 7f35e807c640 1 --2- 172.21.15.170:0/226655779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e0144f50 0x7f35e0145400 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0097d0 tx=0x7f35dc02b8a0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.137+0000 7f35e807c640 1 -- 172.21.15.170:0/226655779 shutdown_connections 2023-10-30T17:32:35.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.137+0000 7f35e807c640 1 --2- 172.21.15.170:0/226655779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e0144f50 0x7f35e0145400 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.137+0000 7f35e807c640 1 -- 172.21.15.170:0/226655779 >> 172.21.15.170:0/226655779 conn(0x7f35e0071da0 msgr2=0x7f35e00b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:35.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.137+0000 7f35e807c640 1 -- 172.21.15.170:0/226655779 shutdown_connections 2023-10-30T17:32:35.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.137+0000 7f35e807c640 1 -- 172.21.15.170:0/226655779 wait complete. 2023-10-30T17:32:35.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.137+0000 7f35e807c640 10 monclient: build_initial_monmap 2023-10-30T17:32:35.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.137+0000 7f35e807c640 10 monclient: monmap: 2023-10-30T17:32:35.153 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:35.153 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:35.153 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:35.153 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:35.153 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:35.154 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.154 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:35.154 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:35.154 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:35.154 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 librados: starting msgr at 2023-10-30T17:32:35.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 librados: starting objecter 2023-10-30T17:32:35.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 Processor -- start 2023-10-30T17:32:35.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 -- start start 2023-10-30T17:32:35.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 librados: setting wanted keys 2023-10-30T17:32:35.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 librados: calling monclient init 2023-10-30T17:32:35.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient: init 2023-10-30T17:32:35.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:35.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:32:35.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35e00b7c90 0x7f35e0138b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient(hunting): picked mon.noname-a con 0x7f35e00b7c90 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:35.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e013eb60 0x7f35e0137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient(hunting): picked mon.noname-c con 0x7f35e013eb60 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:35.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35e0144f50 0x7f35e01313c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient(hunting): picked mon.noname-b con 0x7f35e0144f50 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:35.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:35.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e0138620 con 0x7f35e00b7c90 2023-10-30T17:32:35.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:35.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e0138300 con 0x7f35e0144f50 2023-10-30T17:32:35.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:35.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e01409d0 con 0x7f35e013eb60 2023-10-30T17:32:35.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:35.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.138+0000 7f35e807c640 10 monclient(hunting): authenticate will time out at 1337.461842s 2023-10-30T17:32:35.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5617640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e013eb60 0x7f35e0137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5e18640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35e00b7c90 0x7f35e0138b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e6619640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35e0144f50 0x7f35e01313c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5617640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e013eb60 0x7f35e0137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:53148/0 (socket says 172.21.15.170:53148) 2023-10-30T17:32:35.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5617640 1 -- 172.21.15.170:0/826844159 learned_addr learned my addr 172.21.15.170:0/826844159 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:35.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5617640 10 monclient(hunting): get_auth_request con 0x7f35e013eb60 auth_method 0 2023-10-30T17:32:35.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5617640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:35.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5617640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5617640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5e18640 10 monclient(hunting): get_auth_request con 0x7f35e00b7c90 auth_method 0 2023-10-30T17:32:35.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5e18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:35.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5e18640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e5e18640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e6619640 10 monclient(hunting): get_auth_request con 0x7f35e0144f50 auth_method 0 2023-10-30T17:32:35.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e6619640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:35.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e6619640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.139+0000 7f35e6619640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.140+0000 7f35e5617640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:35.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.140+0000 7f35e5617640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:35.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.140+0000 7f35e5617640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:35.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.140+0000 7f35e6619640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:35.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.140+0000 7f35e6619640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:35.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.140+0000 7f35e6619640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:35.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.140+0000 7f35e5e18640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:35.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.140+0000 7f35e5e18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:35.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.140+0000 7f35e5e18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:35.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.142+0000 7f35e5617640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2023-10-30T17:32:35.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.142+0000 7f35e5617640 1 -- 172.21.15.170:0/826844159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35e0144f50 msgr2=0x7f35e01313c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.142+0000 7f35e5617640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35e0144f50 0x7f35e01313c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.142+0000 7f35e5617640 1 -- 172.21.15.170:0/826844159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35e00b7c90 msgr2=0x7f35e0138b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.142+0000 7f35e5617640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35e00b7c90 0x7f35e0138b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.142+0000 7f35e5617640 10 monclient: _finish_hunting 0 2023-10-30T17:32:35.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.142+0000 7f35e5617640 1 monclient: found mon.noname-c 2023-10-30T17:32:35.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.142+0000 7f35e5617640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:35.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.142+0000 7f35e5617640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:35.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.142+0000 7f35e5617640 1 -- 172.21.15.170:0/826844159 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc009400 con 0x7f35e013eb60 2023-10-30T17:32:35.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.144+0000 7f35e5617640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e013eb60 0x7f35e0137dc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f35d000c530 tx=0x7f35d000c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:35.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.144+0000 7f35d67fc640 10 client.?.objecter ms_handle_connect 0x7f35e013eb60 2023-10-30T17:32:35.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.145+0000 7f35d67fc640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:35.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 1 -- 172.21.15.170:0/826844159 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0005ce0 con 0x7f35e013eb60 2023-10-30T17:32:35.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:35.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:35.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 10 monclient: dump: 2023-10-30T17:32:35.169 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:35.170 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:35.170 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.170 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.170 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:35.170 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.171 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:35.171 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:35.171 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:35.171 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 10 monclient: _finish_auth 0 2023-10-30T17:32:35.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 10 monclient: _check_auth_tickets 2023-10-30T17:32:35.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:35.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 1 -- 172.21.15.170:0/826844159 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d0005e40 con 0x7f35e013eb60 2023-10-30T17:32:35.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:35.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 1 -- 172.21.15.170:0/826844159 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d00072b0 con 0x7f35e013eb60 2023-10-30T17:32:35.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35e807c640 5 monclient: authenticate success, global_id 4238 2023-10-30T17:32:35.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35e807c640 10 monclient: _renew_subs 2023-10-30T17:32:35.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35e807c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:35.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35e807c640 1 -- 172.21.15.170:0/826844159 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e01371a0 con 0x7f35e013eb60 2023-10-30T17:32:35.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d7fff640 4 set_mon_vals no callback set 2023-10-30T17:32:35.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:35.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:35.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35d67fc640 10 monclient: dump: 2023-10-30T17:32:35.174 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:35.174 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:35.174 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.175 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.175 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:35.175 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.175 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:35.175 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:35.176 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:35.176 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35e807c640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:35.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35e807c640 10 monclient: _renew_subs 2023-10-30T17:32:35.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35e807c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:35.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35e807c640 1 -- 172.21.15.170:0/826844159 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e0131ae0 con 0x7f35e013eb60 2023-10-30T17:32:35.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.148+0000 7f35e807c640 1 librados: init done 2023-10-30T17:32:35.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.149+0000 7f35e807c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:35.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.149+0000 7f35e807c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:35.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.149+0000 7f35e807c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:35.178 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.149+0000 7f35e807c640 1 -- 172.21.15.170:0/826844159 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e0067cd0 con 0x7f35e013eb60 2023-10-30T17:32:35.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.424+0000 7f35d67fc640 1 -- 172.21.15.170:0/826844159 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f35d0007410 con 0x7f35e013eb60 2023-10-30T17:32:35.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.424+0000 7f35d67fc640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f35c406c880 0x7f35c406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.425+0000 7f35e5e18640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f35c406c880 0x7f35c406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.425+0000 7f35e5e18640 10 monclient: get_auth_request con 0x7f35c406c880 auth_method 0 2023-10-30T17:32:35.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.425+0000 7f35e5e18640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f35c406c880 0x7f35c406ed40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f35d80012e0 tx=0x7f35d8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:35.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.425+0000 7f35d67fc640 10 client.4238.objecter ms_handle_connect 0x7f35c406c880 2023-10-30T17:32:35.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.438+0000 7f35d67fc640 1 -- 172.21.15.170:0/826844159 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f35d0057eb0 con 0x7f35e013eb60 2023-10-30T17:32:35.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.438+0000 7f35d67fc640 10 client.4238.objecter ms_dispatch 0x7f35e0000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T17:32:35.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.438+0000 7f35d67fc640 3 client.4238.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T17:32:35.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.438+0000 7f35d67fc640 3 client.4238.objecter handle_osd_map decoding full epoch 22 2023-10-30T17:32:35.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.438+0000 7f35d67fc640 20 client.4238.objecter dump_active .. 0 homeless 2023-10-30T17:32:35.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.438+0000 7f35d67fc640 1 -- 172.21.15.170:0/826844159 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f35d00541f0 con 0x7f35e013eb60 2023-10-30T17:32:35.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.438+0000 7f35d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:35.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.438+0000 7f35d67fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:35.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.692+0000 7f35e807c640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-10-30T17:32:35.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.692+0000 7f35e807c640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-10-30T17:32:35.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.692+0000 7f35e807c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:35.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.692+0000 7f35e807c640 1 -- 172.21.15.170:0/826844159 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f35e013e1d0 con 0x7f35e013eb60 2023-10-30T17:32:35.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.824+0000 7f35d67fc640 1 -- 172.21.15.170:0/826844159 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171037 (secure 0 0 0) 0x7f35d0057810 con 0x7f35e013eb60 2023-10-30T17:32:35.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.824+0000 7f35d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-10-30T17:32:35.826 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.824+0000 7f35d67fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:35.828 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:32:35.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.826+0000 7f35e807c640 10 librados: watch_flush enter 2023-10-30T17:32:35.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 10 librados: watch_flush exit 2023-10-30T17:32:35.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 20 client.4238.objecter shutdown clearing up homeless session... 2023-10-30T17:32:35.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 10 client.4238.objecter successfully canceled tick 2023-10-30T17:32:35.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 -- 172.21.15.170:0/826844159 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f35c406c880 msgr2=0x7f35c406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f35c406c880 0x7f35c406ed40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f35d80012e0 tx=0x7f35d8009040 comp rx=0 tx=0).stop 2023-10-30T17:32:35.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 10 monclient: shutdown 2023-10-30T17:32:35.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:35.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 -- 172.21.15.170:0/826844159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e013eb60 msgr2=0x7f35e0137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e013eb60 0x7f35e0137dc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f35d000c530 tx=0x7f35d000c9f0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 -- 172.21.15.170:0/826844159 shutdown_connections 2023-10-30T17:32:35.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f35c406c880 0x7f35c406ed40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35e0144f50 0x7f35e01313c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35e013eb60 0x7f35e0137dc0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 --2- 172.21.15.170:0/826844159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35e00b7c90 0x7f35e0138b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 -- 172.21.15.170:0/826844159 >> 172.21.15.170:0/826844159 conn(0x7f35e0071da0 msgr2=0x7f35e00b6730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:35.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 -- 172.21.15.170:0/826844159 shutdown_connections 2023-10-30T17:32:35.834 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 -- 172.21.15.170:0/826844159 wait complete. 2023-10-30T17:32:35.834 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.827+0000 7f35e807c640 1 librados: shutdown 2023-10-30T17:32:35.840 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":5,"active_gid":4111,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1894191478},{"type":"v1","addr":"172.21.15.177:6801","nonce":1894191478}]},"active_addr":"172.21.15.177:6801/1894191478","active_change":"2023-10-30T17:31:44.490121+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4102,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":4183356486}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":2929021244}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":1699124896}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":3505836920}]}]} 2023-10-30T17:32:35.847 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T17:32:35.848 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T17:32:35.848 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T17:32:35.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.949+0000 7f121ff0f640 10 monclient: get_monmap_and_config 2023-10-30T17:32:35.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.949+0000 7f121ff0f640 10 monclient: build_initial_monmap 2023-10-30T17:32:35.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.949+0000 7f121ff0f640 10 monclient: monmap: 2023-10-30T17:32:35.951 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:35.951 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:35.951 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:35.950167+0000 2023-10-30T17:32:35.952 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:35.950167+0000 2023-10-30T17:32:35.952 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:35.952 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.952 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:35.952 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:35.953 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:35.953 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:35.953 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:35.953 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.949+0000 7f121ff0f640 1 Processor -- start 2023-10-30T17:32:35.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 1 -- start start 2023-10-30T17:32:35.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient: init 2023-10-30T17:32:35.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:35.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-10-30T17:32:35.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f12180b7c90 0x7f12180a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient(hunting): picked mon.noname-c con 0x7f12180b7c90 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:35.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f121813eb60 0x7f121813efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient(hunting): picked mon.noname-b con 0x7f121813eb60 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:35.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1218144f50 0x7f12181453c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient(hunting): picked mon.noname-a con 0x7f1218144f50 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:35.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:35.959 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12180a9500 con 0x7f1218144f50 2023-10-30T17:32:35.959 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:35.959 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12180b8070 con 0x7f12180b7c90 2023-10-30T17:32:35.959 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:35.959 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12180aa740 con 0x7f121813eb60 2023-10-30T17:32:35.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:35.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.950+0000 7f121ff0f640 10 monclient(hunting): authenticate will time out at 1338.273660s 2023-10-30T17:32:35.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121d4aa640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f121813eb60 0x7f121813efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121dcab640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f12180b7c90 0x7f12180a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121d4aa640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f121813eb60 0x7f121813efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:53152/0 (socket says 172.21.15.170:53152) 2023-10-30T17:32:35.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121d4aa640 1 -- 172.21.15.170:0/2860350716 learned_addr learned my addr 172.21.15.170:0/2860350716 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:35.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121d4aa640 10 monclient(hunting): get_auth_request con 0x7f121813eb60 auth_method 0 2023-10-30T17:32:35.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121d4aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:35.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121d4aa640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121d4aa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121dcab640 10 monclient(hunting): get_auth_request con 0x7f12180b7c90 auth_method 0 2023-10-30T17:32:35.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121dcab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:35.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121dcab640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121dcab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121e4ac640 1 --2- 172.21.15.170:0/2860350716 >> v2:172.21.15.170:3300/0 conn(0x7f1218144f50 0x7f12181453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121d4aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:35.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.951+0000 7f121d4aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:35.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.952+0000 7f121d4aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:35.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.952+0000 7f121dcab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:35.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.952+0000 7f121dcab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:35.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.952+0000 7f121dcab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:35.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.952+0000 7f121e4ac640 10 monclient(hunting): get_auth_request con 0x7f1218144f50 auth_method 0 2023-10-30T17:32:35.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.952+0000 7f121e4ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:35.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.952+0000 7f121e4ac640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.952+0000 7f121e4ac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121e4ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:35.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121e4ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:35.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121e4ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:35.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121d4aa640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2023-10-30T17:32:35.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121d4aa640 1 -- 172.21.15.170:0/2860350716 >> v2:172.21.15.177:3300/0 conn(0x7f12180b7c90 msgr2=0x7f12180a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:35.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121d4aa640 1 --2- 172.21.15.170:0/2860350716 >> v2:172.21.15.177:3300/0 conn(0x7f12180b7c90 0x7f12180a8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121d4aa640 1 -- 172.21.15.170:0/2860350716 >> v2:172.21.15.170:3300/0 conn(0x7f1218144f50 msgr2=0x7f12181453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:35.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121d4aa640 1 --2- 172.21.15.170:0/2860350716 >> v2:172.21.15.170:3300/0 conn(0x7f1218144f50 0x7f12181453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121d4aa640 10 monclient: _finish_hunting 0 2023-10-30T17:32:35.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121d4aa640 1 monclient: found mon.noname-b 2023-10-30T17:32:35.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121d4aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:35.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121d4aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:35.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.953+0000 7f121d4aa640 1 -- 172.21.15.170:0/2860350716 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12180aa9b0 con 0x7f121813eb60 2023-10-30T17:32:35.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.955+0000 7f121d4aa640 1 --2- 172.21.15.170:0/2860350716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f121813eb60 0x7f121813efb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f12180b8280 tx=0x7f121402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fc619daee7fe2bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:35.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 1 -- 172.21.15.170:0/2860350716 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12140317c0 con 0x7f121813eb60 2023-10-30T17:32:35.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:35.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:35.972 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 10 monclient: dump: 2023-10-30T17:32:35.972 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:35.972 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:35.972 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.972 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.973 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:35.973 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.973 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:35.973 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:35.973 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:35.973 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 10 monclient: _finish_auth 0 2023-10-30T17:32:35.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 10 monclient: _check_auth_tickets 2023-10-30T17:32:35.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:35.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 1 -- 172.21.15.170:0/2860350716 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1214031920 con 0x7f121813eb60 2023-10-30T17:32:35.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:35.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 1 -- 172.21.15.170:0/2860350716 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214031cb0 con 0x7f121813eb60 2023-10-30T17:32:35.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 5 monclient: authenticate success, global_id 4241 2023-10-30T17:32:35.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:35.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 4 set_mon_vals no callback set 2023-10-30T17:32:35.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:35.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:35.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121cca9640 10 monclient: dump: 2023-10-30T17:32:35.976 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:35.976 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:35.976 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.977 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.977 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:35.977 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.977 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:35.977 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:35.978 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:35.978 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 10 monclient: shutdown 2023-10-30T17:32:35.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:35.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 1 -- 172.21.15.170:0/2860350716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f121813eb60 msgr2=0x7f121813efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 1 --2- 172.21.15.170:0/2860350716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f121813eb60 0x7f121813efb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f12180b8280 tx=0x7f121402b860 comp rx=0 tx=0).stop 2023-10-30T17:32:35.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 1 -- 172.21.15.170:0/2860350716 shutdown_connections 2023-10-30T17:32:35.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 1 --2- 172.21.15.170:0/2860350716 >> v2:172.21.15.170:3300/0 conn(0x7f1218144f50 0x7f12181453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 1 --2- 172.21.15.170:0/2860350716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f121813eb60 0x7f121813efb0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 1 --2- 172.21.15.170:0/2860350716 >> v2:172.21.15.177:3300/0 conn(0x7f12180b7c90 0x7f12180a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 1 -- 172.21.15.170:0/2860350716 >> 172.21.15.170:0/2860350716 conn(0x7f1218071da0 msgr2=0x7f12180b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:35.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.960+0000 7f121ff0f640 1 -- 172.21.15.170:0/2860350716 shutdown_connections 2023-10-30T17:32:35.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.961+0000 7f121ff0f640 1 -- 172.21.15.170:0/2860350716 wait complete. 2023-10-30T17:32:35.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.961+0000 7f121ff0f640 10 monclient: build_initial_monmap 2023-10-30T17:32:35.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.961+0000 7f121ff0f640 10 monclient: monmap: 2023-10-30T17:32:35.980 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:35.981 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:35.981 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:35.981 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:35.981 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:35.981 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.982 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:35.982 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:35.982 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:35.982 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.961+0000 7f121ff0f640 1 librados: starting msgr at 2023-10-30T17:32:35.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.961+0000 7f121ff0f640 1 librados: starting objecter 2023-10-30T17:32:35.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 1 Processor -- start 2023-10-30T17:32:35.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 1 -- start start 2023-10-30T17:32:35.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 1 librados: setting wanted keys 2023-10-30T17:32:35.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 1 librados: calling monclient init 2023-10-30T17:32:35.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient: init 2023-10-30T17:32:35.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:35.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:32:35.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12180b7c90 0x7f12180ab0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient(hunting): picked mon.noname-c con 0x7f12180b7c90 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:35.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f121813eb60 0x7f12181d6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient(hunting): picked mon.noname-a con 0x7f121813eb60 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:35.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1218144f50 0x7f12181d6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:35.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient(hunting): picked mon.noname-b con 0x7f1218144f50 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:35.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:35.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12181391f0 con 0x7f121813eb60 2023-10-30T17:32:35.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:35.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1218138200 con 0x7f1218144f50 2023-10-30T17:32:35.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:35.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12181d72c0 con 0x7f12180b7c90 2023-10-30T17:32:35.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:35.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.962+0000 7f121ff0f640 10 monclient(hunting): authenticate will time out at 1338.285766s 2023-10-30T17:32:35.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121e4ac640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1218144f50 0x7f12181d6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121d4aa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f121813eb60 0x7f12181d6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121e4ac640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1218144f50 0x7f12181d6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60398/0 (socket says 172.21.15.170:60398) 2023-10-30T17:32:35.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121e4ac640 1 -- 172.21.15.170:0/1710764487 learned_addr learned my addr 172.21.15.170:0/1710764487 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:35.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121e4ac640 10 monclient(hunting): get_auth_request con 0x7f1218144f50 auth_method 0 2023-10-30T17:32:35.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121e4ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:35.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121e4ac640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121e4ac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121d4aa640 10 monclient(hunting): get_auth_request con 0x7f121813eb60 auth_method 0 2023-10-30T17:32:35.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121d4aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:35.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121d4aa640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.963+0000 7f121d4aa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121e4ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:35.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121e4ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:35.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121e4ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:35.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121d4aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:35.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121d4aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:35.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121d4aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:35.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121dcab640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12180b7c90 0x7f12180ab0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:35.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121dcab640 10 monclient(hunting): get_auth_request con 0x7f12180b7c90 auth_method 0 2023-10-30T17:32:35.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121dcab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:35.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121dcab640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:35.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.964+0000 7f121dcab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:35.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.965+0000 7f121dcab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:35.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.965+0000 7f121dcab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:35.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.965+0000 7f121dcab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:35.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.966+0000 7f121e4ac640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2023-10-30T17:32:35.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.966+0000 7f121e4ac640 1 -- 172.21.15.170:0/1710764487 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12180b7c90 msgr2=0x7f12180ab0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.966+0000 7f121e4ac640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12180b7c90 0x7f12180ab0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.966+0000 7f121e4ac640 1 -- 172.21.15.170:0/1710764487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f121813eb60 msgr2=0x7f12181d6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:35.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.966+0000 7f121e4ac640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f121813eb60 0x7f12181d6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:35.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.966+0000 7f121e4ac640 10 monclient: _finish_hunting 0 2023-10-30T17:32:35.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.966+0000 7f121e4ac640 1 monclient: found mon.noname-b 2023-10-30T17:32:35.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.966+0000 7f121e4ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:35.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.966+0000 7f121e4ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:35.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.966+0000 7f121e4ac640 1 -- 172.21.15.170:0/1710764487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12140094d0 con 0x7f1218144f50 2023-10-30T17:32:35.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.968+0000 7f121e4ac640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1218144f50 0x7f12181d6d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f121000efc0 tx=0x7f121000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:35.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.968+0000 7f120e7fc640 10 client.?.objecter ms_handle_connect 0x7f1218144f50 2023-10-30T17:32:35.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.968+0000 7f120e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:35.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 1 -- 172.21.15.170:0/1710764487 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1210009ba0 con 0x7f1218144f50 2023-10-30T17:32:35.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:35.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:35.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 10 monclient: dump: 2023-10-30T17:32:35.997 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:35.997 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:35.997 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.997 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:35.998 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:35.998 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:35.998 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:35.998 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:35.998 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:35.999 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:35.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 10 monclient: _finish_auth 0 2023-10-30T17:32:35.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 10 monclient: _check_auth_tickets 2023-10-30T17:32:35.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:35.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 1 -- 172.21.15.170:0/1710764487 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121000f040 con 0x7f1218144f50 2023-10-30T17:32:35.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:36.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 1 -- 172.21.15.170:0/1710764487 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12100146c0 con 0x7f1218144f50 2023-10-30T17:32:36.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:36.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:36.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120e7fc640 10 monclient: dump: 2023-10-30T17:32:36.000 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:36.001 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:36.001 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:36.001 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:36.001 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:36.001 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:36.002 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:36.002 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:36.002 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:36.002 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:36.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f121ff0f640 5 monclient: authenticate success, global_id 4273 2023-10-30T17:32:36.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f121ff0f640 10 monclient: _renew_subs 2023-10-30T17:32:36.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f121ff0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:36.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f121ff0f640 1 -- 172.21.15.170:0/1710764487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121813b6a0 con 0x7f1218144f50 2023-10-30T17:32:36.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f120ffff640 4 set_mon_vals no callback set 2023-10-30T17:32:36.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f121ff0f640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:36.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f121ff0f640 10 monclient: _renew_subs 2023-10-30T17:32:36.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f121ff0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:36.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f121ff0f640 1 -- 172.21.15.170:0/1710764487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121813bf10 con 0x7f1218144f50 2023-10-30T17:32:36.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.972+0000 7f121ff0f640 1 librados: init done 2023-10-30T17:32:36.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.973+0000 7f121ff0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:36.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.973+0000 7f121ff0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:36.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.973+0000 7f121ff0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:36.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.973+0000 7f121ff0f640 1 -- 172.21.15.170:0/1710764487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1218067cd0 con 0x7f1218144f50 2023-10-30T17:32:36.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.991+0000 7f120e7fc640 1 -- 172.21.15.170:0/1710764487 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1210014820 con 0x7f1218144f50 2023-10-30T17:32:36.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.992+0000 7f120e7fc640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f11fc06c850 0x7f11fc06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:36.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.992+0000 7f121dcab640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f11fc06c850 0x7f11fc06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:36.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.992+0000 7f121dcab640 10 monclient: get_auth_request con 0x7f11fc06c850 auth_method 0 2023-10-30T17:32:36.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.992+0000 7f120e7fc640 10 client.4273.objecter ms_handle_connect 0x7f11fc06c850 2023-10-30T17:32:36.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.992+0000 7f121dcab640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f11fc06c850 0x7f11fc06ed10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1208009ac0 tx=0x7f1208009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:36.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.994+0000 7f120e7fc640 1 -- 172.21.15.170:0/1710764487 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f12100595b0 con 0x7f1218144f50 2023-10-30T17:32:36.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.994+0000 7f120e7fc640 10 client.4273.objecter ms_dispatch 0x7f1218000b90 osd_map(22..22 src has 1..22) v4 2023-10-30T17:32:36.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.994+0000 7f120e7fc640 3 client.4273.objecter handle_osd_map got epochs [22,22] > 0 2023-10-30T17:32:36.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.994+0000 7f120e7fc640 3 client.4273.objecter handle_osd_map decoding full epoch 22 2023-10-30T17:32:36.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:35.994+0000 7f120e7fc640 20 client.4273.objecter dump_active .. 0 homeless 2023-10-30T17:32:36.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:36.219+0000 7f120e7fc640 1 -- 172.21.15.170:0/1710764487 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f12100558f0 con 0x7f1218144f50 2023-10-30T17:32:36.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:36.219+0000 7f120e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:36.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:36.219+0000 7f120e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:36.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:36.398+0000 7f121ff0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:36.400 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:36.398+0000 7f121ff0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:36.400 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:36.398+0000 7f121ff0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:36.400 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:36.398+0000 7f121ff0f640 1 -- 172.21.15.170:0/1710764487 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1218066100 con 0x7f1218144f50 2023-10-30T17:32:36.708 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:01:20.674 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:36.820 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:01:20.786 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:36.824 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:01:20.790 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:36.853 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:20.792 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:36.877 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:01:20.843 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:36.904 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:20.843 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:37.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.011+0000 7f120e7fc640 1 -- 172.21.15.170:0/1710764487 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14686 (secure 0 0 0) 0x7f1210018070 con 0x7f1218144f50 2023-10-30T17:32:37.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.011+0000 7f120e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:37.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.011+0000 7f120e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:37.014 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:32:37.014 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":23,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:32:36.202354+0000","last_up_change":"2023-10-30T17:32:27.945525+0000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T17:32:25.610766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T17:32:32.227425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6809","nonce":500152820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6811","nonce":500152820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6815","nonce":500152820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6813","nonce":500152820}]},"public_addr":"172.21.15.170:6809/500152820","cluster_addr":"172.21.15.170:6811/500152820","heartbeat_back_addr":"172.21.15.170:6815/500152820","heartbeat_front_addr":"172.21.15.170:6813/500152820","state":["exists","up"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6825","nonce":527201623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6827","nonce":527201623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6831","nonce":527201623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6829","nonce":527201623}]},"public_addr":"172.21.15.170:6825/527201623","cluster_addr":"172.21.15.170:6827/527201623","heartbeat_back_addr":"172.21.15.170:6831/527201623","heartbeat_front_addr":"172.21.15.170:6829/527201623","state":["exists","up"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6817","nonce":425342180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6819","nonce":425342180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6823","nonce":425342180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6821","nonce":425342180}]},"public_addr":"172.21.15.170:6817/425342180","cluster_addr":"172.21.15.170:6819/425342180","heartbeat_back_addr":"172.21.15.170:6823/425342180","heartbeat_front_addr":"172.21.15.170:6821/425342180","state":["exists","up"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6801","nonce":3657546070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6803","nonce":3657546070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6807","nonce":3657546070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6805","nonce":3657546070}]},"public_addr":"172.21.15.170:6801/3657546070","cluster_addr":"172.21.15.170:6803/3657546070","heartbeat_back_addr":"172.21.15.170:6807/3657546070","heartbeat_front_addr":"172.21.15.170:6805/3657546070","state":["exists","up"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6803","nonce":2107882913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6805","nonce":2107882913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6809","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6811","nonce":2107882913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6807","nonce":2107882913}]},"public_addr":"172.21.15.177:6803/2107882913","cluster_addr":"172.21.15.177:6805/2107882913","heartbeat_back_addr":"172.21.15.177:6811/2107882913","heartbeat_front_addr":"172.21.15.177:6807/2107882913","state":["exists","up"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6810","nonce":2243649533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6813","nonce":2243649533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6817","nonce":2243649533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6815","nonce":2243649533}]},"public_addr":"172.21.15.177:6810/2243649533","cluster_addr":"172.21.15.177:6813/2243649533","heartbeat_back_addr":"172.21.15.177:6817/2243649533","heartbeat_front_addr":"172.21.15.177:6815/2243649533","state":["exists","up"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6827","nonce":1041798931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6829","nonce":1041798931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6833","nonce":1041798931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6831","nonce":1041798931}]},"public_addr":"172.21.15.177:6827/1041798931","cluster_addr":"172.21.15.177:6829/1041798931","heartbeat_back_addr":"172.21.15.177:6833/1041798931","heartbeat_front_addr":"172.21.15.177:6831/1041798931","state":["exists","up"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6819","nonce":3145923709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6821","nonce":3145923709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6825","nonce":3145923709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6823","nonce":3145923709}]},"public_addr":"172.21.15.177:6819/3145923709","cluster_addr":"172.21.15.177:6821/3145923709","heartbeat_back_addr":"172.21.15.177:6825/3145923709","heartbeat_front_addr":"172.21.15.177:6823/3145923709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:24.231313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:24.066267+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:23.315345+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:23.639773+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.166624+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.275780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.364855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:22.235136+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:32:37.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.013+0000 7f121ff0f640 10 librados: watch_flush enter 2023-10-30T17:32:37.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.013+0000 7f121ff0f640 10 librados: watch_flush exit 2023-10-30T17:32:37.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.013+0000 7f121ff0f640 20 client.4273.objecter shutdown clearing up homeless session... 2023-10-30T17:32:37.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.013+0000 7f121ff0f640 10 client.4273.objecter successfully canceled tick 2023-10-30T17:32:37.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.013+0000 7f121ff0f640 1 -- 172.21.15.170:0/1710764487 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f11fc06c850 msgr2=0x7f11fc06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:37.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.013+0000 7f121ff0f640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f11fc06c850 0x7f11fc06ed10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1208009ac0 tx=0x7f1208009190 comp rx=0 tx=0).stop 2023-10-30T17:32:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 10 monclient: shutdown 2023-10-30T17:32:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 -- 172.21.15.170:0/1710764487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1218144f50 msgr2=0x7f12181d6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1218144f50 0x7f12181d6d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f121000efc0 tx=0x7f121000c490 comp rx=0 tx=0).stop 2023-10-30T17:32:37.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 -- 172.21.15.170:0/1710764487 shutdown_connections 2023-10-30T17:32:37.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f11fc06c850 0x7f11fc06ed10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1218144f50 0x7f12181d6d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f121813eb60 0x7f12181d6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 --2- 172.21.15.170:0/1710764487 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12180b7c90 0x7f12180ab0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 -- 172.21.15.170:0/1710764487 >> 172.21.15.170:0/1710764487 conn(0x7f1218071da0 msgr2=0x7f12180b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 -- 172.21.15.170:0/1710764487 shutdown_connections 2023-10-30T17:32:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 -- 172.21.15.170:0/1710764487 wait complete. 2023-10-30T17:32:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.014+0000 7f121ff0f640 1 librados: shutdown 2023-10-30T17:32:37.027 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T17:32:37.027 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T17:32:37.094 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:01:21.060 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:37.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.130+0000 7f3880941640 10 monclient: get_monmap_and_config 2023-10-30T17:32:37.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.130+0000 7f3880941640 10 monclient: build_initial_monmap 2023-10-30T17:32:37.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.130+0000 7f3880941640 10 monclient: monmap: 2023-10-30T17:32:37.132 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:37.132 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:37.132 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:37.131178+0000 2023-10-30T17:32:37.133 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:37.131178+0000 2023-10-30T17:32:37.133 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:37.133 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:37.133 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:37.133 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:37.133 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:37.134 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:37.134 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:37.134 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:37.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.130+0000 7f3880941640 1 Processor -- start 2023-10-30T17:32:37.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.130+0000 7f3880941640 1 -- start start 2023-10-30T17:32:37.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.130+0000 7f3880941640 10 monclient: init 2023-10-30T17:32:37.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:37.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-10-30T17:32:37.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f387c0a8720 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:37.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f387c0a8b00 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:37.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f387c145bf0 0x7f387c14d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:37.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 10 monclient(hunting): picked mon.noname-c con 0x7f387c145bf0 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:37.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387c0a8f80 con 0x7f387c0a8b00 2023-10-30T17:32:37.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387c0a9fd0 con 0x7f387c0a8720 2023-10-30T17:32:37.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:37.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387c0aa240 con 0x7f387c145bf0 2023-10-30T17:32:37.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:37.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.131+0000 7f3880941640 10 monclient(hunting): authenticate will time out at 1339.454994s 2023-10-30T17:32:37.150 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:01:21.116 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:37.181 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:21.120 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:37.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.191+0000 7f387ad9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f387c145bf0 0x7f387c14d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:37.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.196+0000 7f387ad9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f387c145bf0 0x7f387c14d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60408/0 (socket says 172.21.15.170:60408) 2023-10-30T17:32:37.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.197+0000 7f387ad9d640 1 -- 172.21.15.170:0/3077422090 learned_addr learned my addr 172.21.15.170:0/3077422090 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:37.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.197+0000 7f387ad9d640 10 monclient(hunting): get_auth_request con 0x7f387c145bf0 auth_method 0 2023-10-30T17:32:37.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.197+0000 7f387ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:37.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.197+0000 7f387ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:37.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.197+0000 7f387ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.201+0000 7f387ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.201+0000 7f387ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.201+0000 7f387ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:37.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.246+0000 7f387959a640 1 -- 172.21.15.170:0/3077422090 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654768388 0 0) 0x7f387c0a8f80 con 0x7f387c0a8b00 2023-10-30T17:32:37.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.246+0000 7f387959a640 10 monclient(hunting): _negotiate 2023-10-30T17:32:37.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.246+0000 7f387959a640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:37.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.246+0000 7f387959a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:37.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.246+0000 7f387959a640 10 monclient(hunting): my global_id is 4290 2023-10-30T17:32:37.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.246+0000 7f387959a640 1 -- 172.21.15.170:0/3077422090 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3864003520 con 0x7f387c0a8b00 2023-10-30T17:32:37.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.254+0000 7f387ad9d640 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2023-10-30T17:32:37.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.254+0000 7f387ad9d640 1 -- 172.21.15.170:0/3077422090 >> v1:172.21.15.177:6789/0 conn(0x7f387c0a8720 legacy=0x7f387c13ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:37.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.254+0000 7f387ad9d640 1 -- 172.21.15.170:0/3077422090 >> v1:172.21.15.170:6789/0 conn(0x7f387c0a8b00 legacy=0x7f387c1454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:37.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.254+0000 7f387ad9d640 10 monclient: _finish_hunting 0 2023-10-30T17:32:37.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.254+0000 7f387ad9d640 1 monclient: found mon.noname-c 2023-10-30T17:32:37.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.254+0000 7f387ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:37.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.254+0000 7f387ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:32:37.259 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.254+0000 7f387ad9d640 1 -- 172.21.15.170:0/3077422090 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c0aa520 con 0x7f387c145bf0 2023-10-30T17:32:37.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.277+0000 7f387ad9d640 1 --2- 172.21.15.170:0/3077422090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f387c145bf0 0x7f387c14d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3874009e50 tx=0x7f387402bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f73ce7aba23b701 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:37.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 1 -- 172.21.15.170:0/3077422090 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387402cb20 con 0x7f387c145bf0 2023-10-30T17:32:37.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:37.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:37.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 10 monclient: dump: 2023-10-30T17:32:37.308 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:37.308 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:37.308 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:37.308 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:37.308 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:37.309 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:37.309 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:37.309 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:37.309 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:37.309 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:37.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 10 monclient: _finish_auth 0 2023-10-30T17:32:37.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 10 monclient: _check_auth_tickets 2023-10-30T17:32:37.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:37.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 1 -- 172.21.15.170:0/3077422090 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387402cc80 con 0x7f387c145bf0 2023-10-30T17:32:37.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f3880941640 5 monclient: authenticate success, global_id 4279 2023-10-30T17:32:37.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f3880941640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:37.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:37.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f3880941640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:37.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f3880941640 4 set_mon_vals no callback set 2023-10-30T17:32:37.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 1 -- 172.21.15.170:0/3077422090 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38740343f0 con 0x7f387c145bf0 2023-10-30T17:32:37.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:37.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:37.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.304+0000 7f387959a640 10 monclient: dump: 2023-10-30T17:32:37.312 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:37.312 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:37.313 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:37.313 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:37.313 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:37.313 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:37.313 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:37.314 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:37.314 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:37.314 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:37.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.305+0000 7f3880941640 10 monclient: shutdown 2023-10-30T17:32:37.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.305+0000 7f3880941640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:37.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.305+0000 7f3880941640 1 -- 172.21.15.170:0/3077422090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f387c145bf0 msgr2=0x7f387c14d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:37.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.305+0000 7f3880941640 1 --2- 172.21.15.170:0/3077422090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f387c145bf0 0x7f387c14d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3874009e50 tx=0x7f387402bf80 comp rx=0 tx=0).stop 2023-10-30T17:32:37.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.305+0000 7f3880941640 1 -- 172.21.15.170:0/3077422090 shutdown_connections 2023-10-30T17:32:37.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.305+0000 7f3880941640 1 --2- 172.21.15.170:0/3077422090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f387c145bf0 0x7f387c14d3c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.305+0000 7f3880941640 1 -- 172.21.15.170:0/3077422090 >> 172.21.15.170:0/3077422090 conn(0x7f387c0b7300 msgr2=0x7f387c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:37.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.305+0000 7f3880941640 1 -- 172.21.15.170:0/3077422090 shutdown_connections 2023-10-30T17:32:37.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.305+0000 7f3880941640 1 -- 172.21.15.170:0/3077422090 wait complete. 2023-10-30T17:32:37.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.306+0000 7f3880941640 10 monclient: build_initial_monmap 2023-10-30T17:32:37.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.306+0000 7f3880941640 10 monclient: monmap: 2023-10-30T17:32:37.318 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:37.319 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:37.319 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:37.319 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:37.319 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:37.319 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:37.320 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:37.320 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:37.320 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:37.320 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:37.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.306+0000 7f3880941640 1 librados: starting msgr at 2023-10-30T17:32:37.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.306+0000 7f3880941640 1 librados: starting objecter 2023-10-30T17:32:37.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.306+0000 7f3880941640 1 Processor -- start 2023-10-30T17:32:37.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.306+0000 7f3880941640 1 -- start start 2023-10-30T17:32:37.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.306+0000 7f3880941640 1 librados: setting wanted keys 2023-10-30T17:32:37.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.306+0000 7f3880941640 1 librados: calling monclient init 2023-10-30T17:32:37.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.306+0000 7f3880941640 10 monclient: init 2023-10-30T17:32:37.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:37.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:32:37.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f387c0a8720 0x7f387c0b0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:37.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 10 monclient(hunting): picked mon.noname-c con 0x7f387c0a8720 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:37.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f387c0a8b00 0x7f387c144120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:37.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 10 monclient(hunting): picked mon.noname-b con 0x7f387c0a8b00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:37.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f387c145bf0 0x7f387c0af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:37.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 10 monclient(hunting): picked mon.noname-a con 0x7f387c145bf0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:37.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:37.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c0afe60 con 0x7f387c145bf0 2023-10-30T17:32:37.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:37.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c0b0660 con 0x7f387c0a8b00 2023-10-30T17:32:37.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:37.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387c1d6f60 con 0x7f387c0a8720 2023-10-30T17:32:37.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:37.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.307+0000 7f3880941640 10 monclient(hunting): authenticate will time out at 1339.630657s 2023-10-30T17:32:37.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.314+0000 7f387ad9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f387c145bf0 0x7f387c0af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:37.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.315+0000 7f387ad9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f387c145bf0 0x7f387c0af920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:39758/0 (socket says 172.21.15.170:39758) 2023-10-30T17:32:37.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.315+0000 7f387ad9d640 1 -- 172.21.15.170:0/1144402635 learned_addr learned my addr 172.21.15.170:0/1144402635 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:37.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.315+0000 7f387ad9d640 10 monclient(hunting): get_auth_request con 0x7f387c145bf0 auth_method 0 2023-10-30T17:32:37.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.315+0000 7f387ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:37.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.315+0000 7f387ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:37.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.315+0000 7f387ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:37.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.323+0000 7f3879d9b640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f387c0a8b00 0x7f387c144120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:37.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.323+0000 7f387a59c640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f387c0a8720 0x7f387c0b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:37.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.324+0000 7f387a59c640 10 monclient(hunting): get_auth_request con 0x7f387c0a8720 auth_method 0 2023-10-30T17:32:37.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.324+0000 7f387a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:37.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.324+0000 7f387a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:37.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.324+0000 7f387a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:37.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.324+0000 7f387a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:37.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.324+0000 7f387a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:37.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.324+0000 7f387a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:37.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.325+0000 7f3879d9b640 10 monclient(hunting): get_auth_request con 0x7f387c0a8b00 auth_method 0 2023-10-30T17:32:37.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.325+0000 7f3879d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:37.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.325+0000 7f3879d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:37.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.325+0000 7f3879d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:37.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.327+0000 7f3879d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:37.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.327+0000 7f3879d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:37.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.327+0000 7f3879d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.333+0000 7f3879d9b640 10 monclient(hunting): handle_auth_done global_id 4282 payload 824 2023-10-30T17:32:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.333+0000 7f3879d9b640 1 -- 172.21.15.170:0/1144402635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f387c0a8720 msgr2=0x7f387c0b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.333+0000 7f3879d9b640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f387c0a8720 0x7f387c0b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.333+0000 7f3879d9b640 1 -- 172.21.15.170:0/1144402635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f387c145bf0 msgr2=0x7f387c0af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:37.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.333+0000 7f3879d9b640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f387c145bf0 0x7f387c0af920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.333+0000 7f3879d9b640 10 monclient: _finish_hunting 0 2023-10-30T17:32:37.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.333+0000 7f3879d9b640 1 monclient: found mon.noname-b 2023-10-30T17:32:37.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.333+0000 7f3879d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.333+0000 7f3879d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.333+0000 7f3879d9b640 1 -- 172.21.15.170:0/1144402635 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3874009b30 con 0x7f387c0a8b00 2023-10-30T17:32:37.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.343+0000 7f3879d9b640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f387c0a8b00 0x7f387c144120 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f387000c390 tx=0x7f3870001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:37.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.343+0000 7f386affd640 10 client.?.objecter ms_handle_connect 0x7f387c0a8b00 2023-10-30T17:32:37.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.343+0000 7f386affd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:37.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.378+0000 7f386affd640 1 -- 172.21.15.170:0/1144402635 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38700052d0 con 0x7f387c0a8b00 2023-10-30T17:32:37.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.378+0000 7f386affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:37.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.378+0000 7f386affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:37.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.378+0000 7f386affd640 10 monclient: dump: 2023-10-30T17:32:37.382 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:37.382 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:37.382 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:37.382 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:37.382 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:37.383 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:37.383 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:37.383 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:37.383 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:37.383 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:37.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.378+0000 7f386affd640 10 monclient: _finish_auth 0 2023-10-30T17:32:37.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.378+0000 7f386affd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:37.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.378+0000 7f386affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:37.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.378+0000 7f386affd640 1 -- 172.21.15.170:0/1144402635 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3870005430 con 0x7f387c0a8b00 2023-10-30T17:32:37.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.378+0000 7f3880941640 5 monclient: authenticate success, global_id 4282 2023-10-30T17:32:37.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 10 monclient: _renew_subs 2023-10-30T17:32:37.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:37.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 1 -- 172.21.15.170:0/1144402635 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f387c1d7120 con 0x7f387c0a8b00 2023-10-30T17:32:37.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f386affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:37.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f386affd640 1 -- 172.21.15.170:0/1144402635 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870003830 con 0x7f387c0a8b00 2023-10-30T17:32:37.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f386affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:37.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f386affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:37.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f386affd640 10 monclient: dump: 2023-10-30T17:32:37.386 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:37.386 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:37.387 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:37.387 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:37.387 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:37.387 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:37.387 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:37.388 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:37.388 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:37.388 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:37.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3878d99640 4 set_mon_vals no callback set 2023-10-30T17:32:37.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:37.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 10 monclient: _renew_subs 2023-10-30T17:32:37.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:37.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 1 -- 172.21.15.170:0/1144402635 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387c1d7630 con 0x7f387c0a8b00 2023-10-30T17:32:37.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 1 librados: init done 2023-10-30T17:32:37.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:37.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:37.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:37.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.379+0000 7f3880941640 1 -- 172.21.15.170:0/1144402635 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387c13f660 con 0x7f387c0a8b00 2023-10-30T17:32:37.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.687+0000 7f386affd640 1 -- 172.21.15.170:0/1144402635 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3870003990 con 0x7f387c0a8b00 2023-10-30T17:32:37.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.687+0000 7f386affd640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f385c06c800 0x7f385c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:37.690 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.688+0000 7f387a59c640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f385c06c800 0x7f385c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:37.690 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.688+0000 7f387a59c640 10 monclient: get_auth_request con 0x7f385c06c800 auth_method 0 2023-10-30T17:32:37.690 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.688+0000 7f387a59c640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f385c06c800 0x7f385c06ecc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f386c006fd0 tx=0x7f386c009060 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:37.691 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.688+0000 7f386affd640 10 client.4282.objecter ms_handle_connect 0x7f385c06c800 2023-10-30T17:32:37.710 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.708+0000 7f386affd640 1 -- 172.21.15.170:0/1144402635 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3870057dd0 con 0x7f387c0a8b00 2023-10-30T17:32:37.710 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.708+0000 7f386affd640 10 client.4282.objecter ms_dispatch 0x7f387c000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:37.710 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.708+0000 7f386affd640 3 client.4282.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:37.710 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.708+0000 7f386affd640 3 client.4282.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:37.711 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.708+0000 7f386affd640 20 client.4282.objecter dump_active .. 0 homeless 2023-10-30T17:32:37.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.734+0000 7f386affd640 1 -- 172.21.15.170:0/1144402635 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3870054110 con 0x7f387c0a8b00 2023-10-30T17:32:37.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.734+0000 7f386affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:37.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.734+0000 7f386affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:37.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.926+0000 7f3880941640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:37.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.926+0000 7f3880941640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:37.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.926+0000 7f3880941640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:37.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.926+0000 7f3880941640 1 -- 172.21.15.170:0/1144402635 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f387c04e3c0 con 0x7f387c0a8b00 2023-10-30T17:32:37.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.955+0000 7f386affd640 1 -- 172.21.15.170:0/1144402635 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14686 (secure 0 0 0) 0x7f3870017020 con 0x7f387c0a8b00 2023-10-30T17:32:37.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.955+0000 7f386affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:32:37.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.955+0000 7f386affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:37.958 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:32:37.958 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":23,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:32:36.202354+0000","last_up_change":"2023-10-30T17:32:27.945525+0000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T17:32:25.610766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T17:32:32.227425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6809","nonce":500152820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6811","nonce":500152820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6815","nonce":500152820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6813","nonce":500152820}]},"public_addr":"172.21.15.170:6809/500152820","cluster_addr":"172.21.15.170:6811/500152820","heartbeat_back_addr":"172.21.15.170:6815/500152820","heartbeat_front_addr":"172.21.15.170:6813/500152820","state":["exists","up"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6825","nonce":527201623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6827","nonce":527201623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6831","nonce":527201623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6829","nonce":527201623}]},"public_addr":"172.21.15.170:6825/527201623","cluster_addr":"172.21.15.170:6827/527201623","heartbeat_back_addr":"172.21.15.170:6831/527201623","heartbeat_front_addr":"172.21.15.170:6829/527201623","state":["exists","up"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6817","nonce":425342180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6819","nonce":425342180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6823","nonce":425342180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6821","nonce":425342180}]},"public_addr":"172.21.15.170:6817/425342180","cluster_addr":"172.21.15.170:6819/425342180","heartbeat_back_addr":"172.21.15.170:6823/425342180","heartbeat_front_addr":"172.21.15.170:6821/425342180","state":["exists","up"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6801","nonce":3657546070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6803","nonce":3657546070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6807","nonce":3657546070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6805","nonce":3657546070}]},"public_addr":"172.21.15.170:6801/3657546070","cluster_addr":"172.21.15.170:6803/3657546070","heartbeat_back_addr":"172.21.15.170:6807/3657546070","heartbeat_front_addr":"172.21.15.170:6805/3657546070","state":["exists","up"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6803","nonce":2107882913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6805","nonce":2107882913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6809","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6811","nonce":2107882913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6807","nonce":2107882913}]},"public_addr":"172.21.15.177:6803/2107882913","cluster_addr":"172.21.15.177:6805/2107882913","heartbeat_back_addr":"172.21.15.177:6811/2107882913","heartbeat_front_addr":"172.21.15.177:6807/2107882913","state":["exists","up"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6810","nonce":2243649533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6813","nonce":2243649533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6817","nonce":2243649533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6815","nonce":2243649533}]},"public_addr":"172.21.15.177:6810/2243649533","cluster_addr":"172.21.15.177:6813/2243649533","heartbeat_back_addr":"172.21.15.177:6817/2243649533","heartbeat_front_addr":"172.21.15.177:6815/2243649533","state":["exists","up"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6827","nonce":1041798931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6829","nonce":1041798931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6833","nonce":1041798931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6831","nonce":1041798931}]},"public_addr":"172.21.15.177:6827/1041798931","cluster_addr":"172.21.15.177:6829/1041798931","heartbeat_back_addr":"172.21.15.177:6833/1041798931","heartbeat_front_addr":"172.21.15.177:6831/1041798931","state":["exists","up"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6819","nonce":3145923709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6821","nonce":3145923709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6825","nonce":3145923709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6823","nonce":3145923709}]},"public_addr":"172.21.15.177:6819/3145923709","cluster_addr":"172.21.15.177:6821/3145923709","heartbeat_back_addr":"172.21.15.177:6825/3145923709","heartbeat_front_addr":"172.21.15.177:6823/3145923709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:24.231313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:24.066267+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:23.315345+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:23.639773+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.166624+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.275780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.364855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:22.235136+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:32:37.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 10 librados: watch_flush enter 2023-10-30T17:32:37.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 10 librados: watch_flush exit 2023-10-30T17:32:37.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 20 client.4282.objecter shutdown clearing up homeless session... 2023-10-30T17:32:37.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 10 client.4282.objecter successfully canceled tick 2023-10-30T17:32:37.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 -- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f385c06c800 msgr2=0x7f385c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:37.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f385c06c800 0x7f385c06ecc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f386c006fd0 tx=0x7f386c009060 comp rx=0 tx=0).stop 2023-10-30T17:32:37.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 10 monclient: shutdown 2023-10-30T17:32:37.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:37.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 -- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f387c0a8b00 msgr2=0x7f387c144120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:37.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f387c0a8b00 0x7f387c144120 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f387000c390 tx=0x7f3870001e70 comp rx=0 tx=0).stop 2023-10-30T17:32:37.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 -- 172.21.15.170:0/1144402635 shutdown_connections 2023-10-30T17:32:37.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f385c06c800 0x7f385c06ecc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f387c0a8b00 0x7f387c144120 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f387c145bf0 0x7f387c0af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 --2- 172.21.15.170:0/1144402635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f387c0a8720 0x7f387c0b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:37.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 -- 172.21.15.170:0/1144402635 >> 172.21.15.170:0/1144402635 conn(0x7f387c0b7300 msgr2=0x7f387c0b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:37.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 -- 172.21.15.170:0/1144402635 shutdown_connections 2023-10-30T17:32:37.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 -- 172.21.15.170:0/1144402635 wait complete. 2023-10-30T17:32:37.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:37.957+0000 7f3880941640 1 librados: shutdown 2023-10-30T17:32:37.971 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T17:32:37.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T17:32:37.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T17:32:37.973 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T17:32:37.973 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T17:32:37.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T17:32:37.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-30T17:32:37.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-30T17:32:38.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.146+0000 7f9df8e4b640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.146+0000 7f9df8e4b640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.146+0000 7f9df8e4b640 10 monclient: monmap: 2023-10-30T17:32:38.149 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.149 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.149 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.147671+0000 2023-10-30T17:32:38.149 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.147671+0000 2023-10-30T17:32:38.149 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.150 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.150 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.150 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.150 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.150 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.151 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.151 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.147+0000 7f3a0e475640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.147+0000 7f3a0e475640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.147+0000 7f3a0e475640 10 monclient: monmap: 2023-10-30T17:32:38.152 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.152 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.152 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.148239+0000 2023-10-30T17:32:38.152 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.148239+0000 2023-10-30T17:32:38.153 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.153 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.153 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.153 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.153 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.154 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.154 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.154 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.147+0000 7f6049f87640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.147+0000 7f6049f87640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.147+0000 7f6049f87640 10 monclient: monmap: 2023-10-30T17:32:38.155 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.155 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.156 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.148178+0000 2023-10-30T17:32:38.156 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.148178+0000 2023-10-30T17:32:38.156 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.156 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.156 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.156 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.157 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.157 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.157 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.157 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.148+0000 7f9df8e4b640 1 Processor -- start 2023-10-30T17:32:38.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.148+0000 7f6049f87640 1 Processor -- start 2023-10-30T17:32:38.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.148+0000 7f3a0e475640 1 Processor -- start 2023-10-30T17:32:38.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.148+0000 7f9df8e4b640 1 -- start start 2023-10-30T17:32:38.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.148+0000 7f9df8e4b640 10 monclient: init 2023-10-30T17:32:38.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-10-30T17:32:38.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dec039510 0x7f9dec0c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 10 monclient(hunting): picked mon.noname-b con 0x7f9dec039510 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.148+0000 7f3a0e475640 1 -- start start 2023-10-30T17:32:38.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.148+0000 7f3a0e475640 10 monclient: init 2023-10-30T17:32:38.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-10-30T17:32:38.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3a080ae020 0x7f3a080ac480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 10 monclient(hunting): picked mon.noname-c con 0x7f3a080ae020 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:38.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3a080ae400 0x7f3a0813ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 10 monclient(hunting): picked mon.noname-a con 0x7f3a080ae400 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:38.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a0813daf0 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:38.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a080ac9c0 con 0x7f3a0813daf0 2023-10-30T17:32:38.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a080acc30 con 0x7f3a080ae400 2023-10-30T17:32:38.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0813e690 con 0x7f3a080ae020 2023-10-30T17:32:38.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f3a0e475640 10 monclient(hunting): authenticate will time out at 1340.472726s 2023-10-30T17:32:38.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9dec0398f0 0x7f9dec035a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 10 monclient(hunting): picked mon.noname-a con 0x7f9dec0398f0 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:38.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9dec035f60 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:38.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dec0cc180 con 0x7f9dec035f60 2023-10-30T17:32:38.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dec0c7dc0 con 0x7f9dec0398f0 2023-10-30T17:32:38.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec037790 con 0x7f9dec039510 2023-10-30T17:32:38.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.149+0000 7f9df8e4b640 10 monclient(hunting): authenticate will time out at 1340.472732s 2023-10-30T17:32:38.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 1 -- start start 2023-10-30T17:32:38.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 10 monclient: init 2023-10-30T17:32:38.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-10-30T17:32:38.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f60440ae020 0x7f60440ac480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 10 monclient(hunting): picked mon.noname-c con 0x7f60440ae020 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:38.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60440ae400 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:38.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604413daf0 0x7f604413dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 10 monclient(hunting): picked mon.noname-b con 0x7f604413daf0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60440ac9c0 con 0x7f60440ae400 2023-10-30T17:32:38.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60440acc30 con 0x7f60440ae020 2023-10-30T17:32:38.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60440ab770 con 0x7f604413daf0 2023-10-30T17:32:38.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.150+0000 7f6049f87640 10 monclient(hunting): authenticate will time out at 1340.473973s 2023-10-30T17:32:38.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.151+0000 7f9df259c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dec039510 0x7f9dec0c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.151+0000 7f9df1d9b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9dec0398f0 0x7f9dec035a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.151+0000 7f9df1d9b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9dec0398f0 0x7f9dec035a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:39782/0 (socket says 172.21.15.170:39782) 2023-10-30T17:32:38.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.151+0000 7f9df1d9b640 1 -- 172.21.15.170:0/86420166 learned_addr learned my addr 172.21.15.170:0/86420166 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.151+0000 7f9df1d9b640 10 monclient(hunting): get_auth_request con 0x7f9dec0398f0 auth_method 0 2023-10-30T17:32:38.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.151+0000 7f9df1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.151+0000 7f9df1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.151+0000 7f9df1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.151+0000 7f6049786640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604413daf0 0x7f604413dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f9df259c640 10 monclient(hunting): get_auth_request con 0x7f9dec039510 auth_method 0 2023-10-30T17:32:38.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f9df259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f9df259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f9df259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f2c72733640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f2c72733640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f2c72733640 10 monclient: monmap: 2023-10-30T17:32:38.175 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.175 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.176 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.153448+0000 2023-10-30T17:32:38.176 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.153448+0000 2023-10-30T17:32:38.176 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.176 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.176 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.177 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.177 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.177 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.177 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.177 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.178 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f9df1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.178 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f9df1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.178 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f9df1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.178 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.152+0000 7f3a0cc72640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3a080ae400 0x7f3a0813ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.179 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.153+0000 7f3a0d473640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3a080ae020 0x7f3a080ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.179 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.153+0000 7f6049786640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604413daf0 0x7f604413dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:53202/0 (socket says 172.21.15.170:53202) 2023-10-30T17:32:38.179 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.153+0000 7f6049786640 1 -- 172.21.15.170:0/946838169 learned_addr learned my addr 172.21.15.170:0/946838169 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.153+0000 7f6049786640 10 monclient(hunting): get_auth_request con 0x7f604413daf0 auth_method 0 2023-10-30T17:32:38.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.153+0000 7f6049786640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.153+0000 7f6049786640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.153+0000 7f6049786640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.153+0000 7f2c72733640 1 Processor -- start 2023-10-30T17:32:38.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.153+0000 7f2c72733640 1 -- start start 2023-10-30T17:32:38.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.153+0000 7f2c72733640 10 monclient: init 2023-10-30T17:32:38.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f6049786640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f6049786640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f6049786640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-10-30T17:32:38.184 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c6c0b0330 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:38.184 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c6c0a8c00 0x7f2c6c0a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.184 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 10 monclient(hunting): picked mon.noname-b con 0x7f2c6c0a8c00 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.184 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2c6c0a7220 0x7f2c6c13cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.184 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 10 monclient(hunting): picked mon.noname-c con 0x7f2c6c0a7220 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:38.184 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c0a82b0 con 0x7f2c6c0b0330 2023-10-30T17:32:38.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6c0b0710 con 0x7f2c6c0a7220 2023-10-30T17:32:38.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c0aac60 con 0x7f2c6c0a8c00 2023-10-30T17:32:38.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.186 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f2c72733640 10 monclient(hunting): authenticate will time out at 1340.477713s 2023-10-30T17:32:38.186 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f3a0d473640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3a080ae020 0x7f3a080ac480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60416/0 (socket says 172.21.15.170:60416) 2023-10-30T17:32:38.186 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f3a0d473640 1 -- 172.21.15.170:0/829057584 learned_addr learned my addr 172.21.15.170:0/829057584 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.186 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f3a0d473640 10 monclient(hunting): get_auth_request con 0x7f3a080ae020 auth_method 0 2023-10-30T17:32:38.187 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f3a0d473640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.187 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f3a0d473640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.187 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.154+0000 7f3a0d473640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.155+0000 7f9df1d9b640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2023-10-30T17:32:38.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.155+0000 7f9df1d9b640 1 -- 172.21.15.170:0/86420166 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dec039510 msgr2=0x7f9dec0c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.155+0000 7f9df1d9b640 1 --2- 172.21.15.170:0/86420166 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dec039510 0x7f9dec0c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.155+0000 7f9df1d9b640 1 -- 172.21.15.170:0/86420166 >> v1:172.21.15.177:6789/0 conn(0x7f9dec035f60 legacy=0x7f9dec0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.155+0000 7f9df1d9b640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.155+0000 7f9df1d9b640 1 monclient: found mon.noname-a 2023-10-30T17:32:38.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.155+0000 7f9df1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.155+0000 7f9df1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:38.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.155+0000 7f9df1d9b640 1 -- 172.21.15.170:0/86420166 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec0cb2b0 con 0x7f9dec0398f0 2023-10-30T17:32:38.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f3a0cc72640 10 monclient(hunting): get_auth_request con 0x7f3a080ae400 auth_method 0 2023-10-30T17:32:38.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f3a0cc72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f3a0cc72640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f3a0cc72640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f3a0d473640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f3a0d473640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f3a0d473640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f6048f85640 1 --2- 172.21.15.170:0/946838169 >> v2:172.21.15.177:3300/0 conn(0x7f60440ae020 0x7f60440ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f6048f85640 10 monclient(hunting): get_auth_request con 0x7f60440ae020 auth_method 0 2023-10-30T17:32:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f6048f85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f6048f85640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f6048f85640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f6049786640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2023-10-30T17:32:38.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.156+0000 7f6049786640 1 -- 172.21.15.170:0/946838169 >> v2:172.21.15.177:3300/0 conn(0x7f60440ae020 msgr2=0x7f60440ac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.157+0000 7f6049786640 1 --2- 172.21.15.170:0/946838169 >> v2:172.21.15.177:3300/0 conn(0x7f60440ae020 0x7f60440ac480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.157+0000 7f6049786640 1 -- 172.21.15.170:0/946838169 >> v1:172.21.15.177:6789/0 conn(0x7f60440ae400 legacy=0x7f604413ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.157+0000 7f6049786640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.157+0000 7f6049786640 1 monclient: found mon.noname-b 2023-10-30T17:32:38.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.157+0000 7f6049786640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.157+0000 7f6049786640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.157+0000 7f6049786640 1 -- 172.21.15.170:0/946838169 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440aba50 con 0x7f604413daf0 2023-10-30T17:32:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.157+0000 7feeda1ed640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.157+0000 7feeda1ed640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.157+0000 7feeda1ed640 10 monclient: monmap: 2023-10-30T17:32:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.158549+0000 2023-10-30T17:32:38.197 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.158549+0000 2023-10-30T17:32:38.197 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.197 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.197 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.197 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.197 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.198 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.198 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.198 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.158+0000 7f2c6bfff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f2c6c0b0330 0x7f2c6c0a6b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:39544/0 (socket says 172.21.15.170:39544) 2023-10-30T17:32:38.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.158+0000 7f2c6bfff640 1 -- 172.21.15.170:0/504639037 learned_addr learned my addr 172.21.15.170:0/504639037 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:38.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.158+0000 7feeda1ed640 1 Processor -- start 2023-10-30T17:32:38.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.158+0000 7feeda1ed640 1 -- start start 2023-10-30T17:32:38.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.158+0000 7feeda1ed640 10 monclient: init 2023-10-30T17:32:38.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.159+0000 7f3e02b97640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.159+0000 7f3e02b97640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.159+0000 7f3e02b97640 10 monclient: monmap: 2023-10-30T17:32:38.201 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.201 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.201 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.160817+0000 2023-10-30T17:32:38.202 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.160817+0000 2023-10-30T17:32:38.202 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.202 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.202 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.202 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.203 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.203 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.203 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.203 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-10-30T17:32:38.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feecc039510 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:38.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feecc0398f0 0x7feecc035a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 10 monclient(hunting): picked mon.noname-b con 0x7feecc0398f0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7feecc035f60 0x7feecc0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 10 monclient(hunting): picked mon.noname-c con 0x7feecc035f60 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:38.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feecc0c59c0 con 0x7feecc039510 2023-10-30T17:32:38.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feecc0c7dc0 con 0x7feecc035f60 2023-10-30T17:32:38.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7feeda1ed640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.160+0000 7f3e02b97640 1 Processor -- start 2023-10-30T17:32:38.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.161+0000 7feeda1ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feecc0cc180 con 0x7feecc0398f0 2023-10-30T17:32:38.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.161+0000 7feeda1ed640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.161+0000 7f3e02b97640 1 -- start start 2023-10-30T17:32:38.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.161+0000 7f3e02b97640 10 monclient: init 2023-10-30T17:32:38.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.161+0000 7feeda1ed640 10 monclient(hunting): authenticate will time out at 1340.484291s 2023-10-30T17:32:38.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.161+0000 7f6049786640 1 --2- 172.21.15.170:0/946838169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604413daf0 0x7f604413dfa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6034009870 tx=0x7f603402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9e2acc4b9f29267 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-10-30T17:32:38.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3dfc0b7c90 0x7f3dfc0a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 10 monclient(hunting): picked mon.noname-a con 0x7f3dfc0b7c90 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:38.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3dfc13eb60 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:38.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dfc144f50 0x7f3dfc145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 10 monclient(hunting): picked mon.noname-b con 0x7f3dfc144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc0aa740 con 0x7f3dfc13eb60 2023-10-30T17:32:38.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc0a9500 con 0x7f3dfc0b7c90 2023-10-30T17:32:38.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc13f6c0 con 0x7f3dfc144f50 2023-10-30T17:32:38.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f3e02b97640 10 monclient(hunting): authenticate will time out at 1340.485611s 2023-10-30T17:32:38.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.162+0000 7f9df1d9b640 1 --2- 172.21.15.170:0/86420166 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec0398f0 0x7f9dec035a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9df4061be0 tx=0x7f9df40a58e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86326532948eb07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.163+0000 7f3a0d473640 10 monclient(hunting): handle_auth_done global_id 4285 payload 274 2023-10-30T17:32:38.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.163+0000 7f3a0d473640 1 -- 172.21.15.170:0/829057584 >> v2:172.21.15.170:3300/0 conn(0x7f3a080ae400 msgr2=0x7f3a0813ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.163+0000 7f3a0d473640 1 --2- 172.21.15.170:0/829057584 >> v2:172.21.15.170:3300/0 conn(0x7f3a080ae400 0x7f3a0813ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.163+0000 7f3a0d473640 1 -- 172.21.15.170:0/829057584 >> v1:172.21.15.177:6789/0 conn(0x7f3a0813daf0 legacy=0x7f3a0813df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.163+0000 7f3a0d473640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.163+0000 7f3a0d473640 1 monclient: found mon.noname-c 2023-10-30T17:32:38.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.163+0000 7f3a0d473640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.163+0000 7f3a0d473640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.163+0000 7f3a0d473640 1 -- 172.21.15.170:0/829057584 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a080ab770 con 0x7f3a080ae020 2023-10-30T17:32:38.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 1 -- 172.21.15.170:0/946838169 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60340317c0 con 0x7f604413daf0 2023-10-30T17:32:38.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:38.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 10 monclient: dump: 2023-10-30T17:32:38.218 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.218 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.218 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.218 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.219 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.219 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.219 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.219 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.219 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.220 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f6049f87640 5 monclient: authenticate success, global_id 4253 2023-10-30T17:32:38.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f6049f87640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:38.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 1 -- 172.21.15.170:0/946838169 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034031920 con 0x7f604413daf0 2023-10-30T17:32:38.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f6049f87640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:38.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f6049f87640 4 set_mon_vals no callback set 2023-10-30T17:32:38.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 1 -- 172.21.15.170:0/946838169 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034031cb0 con 0x7f604413daf0 2023-10-30T17:32:38.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:38.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7f60437fe640 10 monclient: dump: 2023-10-30T17:32:38.222 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.222 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.223 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.223 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.223 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.223 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.223 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.224 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.224 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.224 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.166+0000 7feed99ec640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7feecc035f60 0x7feecc0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f6049f87640 10 monclient: shutdown 2023-10-30T17:32:38.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f6049f87640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f6049f87640 1 -- 172.21.15.170:0/946838169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604413daf0 msgr2=0x7f604413dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f6049f87640 1 --2- 172.21.15.170:0/946838169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604413daf0 0x7f604413dfa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6034009870 tx=0x7f603402b860 comp rx=0 tx=0).stop 2023-10-30T17:32:38.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f6049f87640 1 -- 172.21.15.170:0/946838169 shutdown_connections 2023-10-30T17:32:38.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f6049f87640 1 --2- 172.21.15.170:0/946838169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604413daf0 0x7f604413dfa0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f2c6b7fe640 1 --2- 172.21.15.170:0/504639037 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c6c0a8c00 0x7f2c6c0a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f6049f87640 1 --2- 172.21.15.170:0/946838169 >> v2:172.21.15.177:3300/0 conn(0x7f60440ae020 0x7f60440ac480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f6049f87640 1 -- 172.21.15.170:0/946838169 >> 172.21.15.170:0/946838169 conn(0x7f60440afbe0 msgr2=0x7f60440afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f6049f87640 1 -- 172.21.15.170:0/946838169 shutdown_connections 2023-10-30T17:32:38.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.167+0000 7f6049f87640 1 -- 172.21.15.170:0/946838169 wait complete. 2023-10-30T17:32:38.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f2c70cd0640 1 --2- 172.21.15.170:0/504639037 >> v2:172.21.15.177:3300/0 conn(0x7f2c6c0a7220 0x7f2c6c13cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7feed99ec640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7feecc035f60 0x7feecc0c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60456/0 (socket says 172.21.15.170:60456) 2023-10-30T17:32:38.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7feed99ec640 1 -- 172.21.15.170:0/3866374398 learned_addr learned my addr 172.21.15.170:0/3866374398 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7feed99ec640 10 monclient(hunting): get_auth_request con 0x7feecc035f60 auth_method 0 2023-10-30T17:32:38.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7feed99ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7feed99ec640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7feed99ec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f6049f87640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f6049f87640 10 monclient: monmap: 2023-10-30T17:32:38.230 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.230 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:38.231 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:38.231 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:38.231 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.231 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.231 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:38.232 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:38.232 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:38.232 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f6049f87640 1 librados: starting msgr at 2023-10-30T17:32:38.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f6049f87640 1 librados: starting objecter 2023-10-30T17:32:38.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f2c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f2c6c0a8c00 auth_method 0 2023-10-30T17:32:38.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f2c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f2c6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f2c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.234 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f6049f87640 1 Processor -- start 2023-10-30T17:32:38.234 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f6049f87640 1 -- start start 2023-10-30T17:32:38.234 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f6049f87640 1 librados: setting wanted keys 2023-10-30T17:32:38.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f6049f87640 1 librados: calling monclient init 2023-10-30T17:32:38.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7f6049f87640 10 monclient: init 2023-10-30T17:32:38.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.168+0000 7feed89ea640 1 --2- 172.21.15.170:0/3866374398 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feecc0398f0 0x7feecc035a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f2c70cd0640 10 monclient(hunting): get_auth_request con 0x7f2c6c0a7220 auth_method 0 2023-10-30T17:32:38.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f2c70cd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f2c70cd0640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f2c70cd0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.237 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7feed99ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.237 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7feed99ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.237 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7feed99ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:32:38.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60440ae020 0x7f6044135620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 10 monclient(hunting): picked mon.noname-b con 0x7f60440ae020 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:38.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f604413daf0 0x7f60440707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 10 monclient(hunting): picked mon.noname-a con 0x7f604413daf0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:38.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6044070e00 0x7f60441da4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 10 monclient(hunting): picked mon.noname-c con 0x7f6044070e00 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604406fe10 con 0x7f604413daf0 2023-10-30T17:32:38.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440a84b0 con 0x7f60440ae020 2023-10-30T17:32:38.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6044071f90 con 0x7f6044070e00 2023-10-30T17:32:38.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f6049f87640 10 monclient(hunting): authenticate will time out at 1340.492741s 2023-10-30T17:32:38.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 1 -- 172.21.15.170:0/86420166 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df40ab700 con 0x7f9dec0398f0 2023-10-30T17:32:38.242 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.242 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7feed89ea640 10 monclient(hunting): get_auth_request con 0x7feecc0398f0 auth_method 0 2023-10-30T17:32:38.242 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7feed89ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.242 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7feed89ea640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.243 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7feed89ea640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:38.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 10 monclient: dump: 2023-10-30T17:32:38.244 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.244 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.245 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.245 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.245 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.245 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.245 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.246 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.246 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.246 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 1 -- 172.21.15.170:0/86420166 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df40ab8a0 con 0x7f9dec0398f0 2023-10-30T17:32:38.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 1 -- 172.21.15.170:0/86420166 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df40abcd0 con 0x7f9dec0398f0 2023-10-30T17:32:38.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:38.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.169+0000 7f9df159a640 10 monclient: dump: 2023-10-30T17:32:38.248 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.248 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.248 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.248 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.249 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.249 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.249 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.249 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.249 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.250 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f9df8e4b640 5 monclient: authenticate success, global_id 4296 2023-10-30T17:32:38.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f9df8e4b640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:38.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f9df8e4b640 4 set_mon_vals no callback set 2023-10-30T17:32:38.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f2c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f2c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f2c6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f2c70cd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f2c70cd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f2c70cd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f3dfbfff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f3dfc13eb60 0x7f3dfc13efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:39546/0 (socket says 172.21.15.170:39546) 2023-10-30T17:32:38.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f3dfbfff640 1 -- 172.21.15.170:0/2081714406 learned_addr learned my addr 172.21.15.170:0/2081714406 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:38.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7feed89ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7feed89ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7feed89ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f9df8e4b640 10 monclient: shutdown 2023-10-30T17:32:38.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f9df8e4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f9df8e4b640 1 -- 172.21.15.170:0/86420166 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec0398f0 msgr2=0x7f9dec035a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.170+0000 7f9df8e4b640 1 --2- 172.21.15.170:0/86420166 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec0398f0 0x7f9dec035a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9df4061be0 tx=0x7f9df40a58e0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f3e01134640 1 --2- 172.21.15.170:0/2081714406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dfc144f50 0x7f3dfc145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f9df8e4b640 1 -- 172.21.15.170:0/86420166 shutdown_connections 2023-10-30T17:32:38.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f9df8e4b640 1 --2- 172.21.15.170:0/86420166 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec0398f0 0x7f9dec035a20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f9df8e4b640 1 --2- 172.21.15.170:0/86420166 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dec039510 0x7f9dec0c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f9df8e4b640 1 -- 172.21.15.170:0/86420166 >> 172.21.15.170:0/86420166 conn(0x7f9dec0070a0 msgr2=0x7f9dec037df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f9df8e4b640 1 -- 172.21.15.170:0/86420166 shutdown_connections 2023-10-30T17:32:38.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f9df8e4b640 1 -- 172.21.15.170:0/86420166 wait complete. 2023-10-30T17:32:38.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f9496d83640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f9496d83640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f9496d83640 10 monclient: monmap: 2023-10-30T17:32:38.258 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.258 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.258 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.172407+0000 2023-10-30T17:32:38.258 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.172407+0000 2023-10-30T17:32:38.258 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.259 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.259 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.259 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.259 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.259 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.260 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.260 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f3e01134640 10 monclient(hunting): get_auth_request con 0x7f3dfc144f50 auth_method 0 2023-10-30T17:32:38.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f3e01134640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f3e01134640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.171+0000 7f3e01134640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7feed99ec640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2023-10-30T17:32:38.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7feed99ec640 1 -- 172.21.15.170:0/3866374398 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feecc0398f0 msgr2=0x7feecc035a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7feed99ec640 1 --2- 172.21.15.170:0/3866374398 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feecc0398f0 0x7feecc035a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7feed99ec640 1 -- 172.21.15.170:0/3866374398 >> v1:172.21.15.170:6789/0 conn(0x7feecc039510 legacy=0x7feecc0c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7feed99ec640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7feed99ec640 1 monclient: found mon.noname-c 2023-10-30T17:32:38.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7feed99ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7feed99ec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7feed99ec640 1 -- 172.21.15.170:0/3866374398 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc037790 con 0x7feecc035f60 2023-10-30T17:32:38.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7f9df8e4b640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7f9df8e4b640 10 monclient: monmap: 2023-10-30T17:32:38.264 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.264 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:38.264 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:38.265 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:38.265 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.265 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.265 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:38.265 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:38.266 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:38.266 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7f9496d83640 1 Processor -- start 2023-10-30T17:32:38.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7f9df8e4b640 1 librados: starting msgr at 2023-10-30T17:32:38.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7f9df8e4b640 1 librados: starting objecter 2023-10-30T17:32:38.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7f9496d83640 1 -- start start 2023-10-30T17:32:38.268 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.172+0000 7f9496d83640 10 monclient: init 2023-10-30T17:32:38.268 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f9df8e4b640 1 Processor -- start 2023-10-30T17:32:38.268 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f3a0d473640 1 --2- 172.21.15.170:0/829057584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a080ae020 0x7f3a080ac480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3a0806ae10 tx=0x7f3a0402df20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4cb7b0422c805eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f9df8e4b640 1 -- start start 2023-10-30T17:32:38.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f9df8e4b640 1 librados: setting wanted keys 2023-10-30T17:32:38.270 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f9df8e4b640 1 librados: calling monclient init 2023-10-30T17:32:38.270 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f9df8e4b640 10 monclient: init 2023-10-30T17:32:38.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f9496d83640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f9496d83640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-10-30T17:32:38.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f9496d83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9490070100 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:38.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f9496d83640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94900704e0 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:38.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.173+0000 7f9496d83640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f949013d040 0x7f94900afae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9496d83640 10 monclient(hunting): picked mon.noname-a con 0x7f949013d040 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:38.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9496d83640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94900714e0 con 0x7f94900704e0 2023-10-30T17:32:38.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9496d83640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9490070f30 con 0x7f9490070100 2023-10-30T17:32:38.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9496d83640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9496d83640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9490140720 con 0x7f949013d040 2023-10-30T17:32:38.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9496d83640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9496d83640 10 monclient(hunting): authenticate will time out at 1340.497131s 2023-10-30T17:32:38.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T17:32:38.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dec035f60 0x7f9dec02e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 10 monclient(hunting): picked mon.noname-b con 0x7f9dec035f60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:38.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dec039510 0x7f9dec02fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 10 monclient(hunting): picked mon.noname-c con 0x7f9dec039510 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec0bbe00 0x7f9dec0301d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 10 monclient(hunting): picked mon.noname-a con 0x7f9dec0bbe00 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:38.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec009ff0 con 0x7f9dec0bbe00 2023-10-30T17:32:38.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec00a2d0 con 0x7f9dec035f60 2023-10-30T17:32:38.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec009ca0 con 0x7f9dec039510 2023-10-30T17:32:38.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.174+0000 7f9df8e4b640 10 monclient(hunting): authenticate will time out at 1340.497577s 2023-10-30T17:32:38.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.175+0000 7f3e01134640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.175+0000 7f3e01134640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.175+0000 7f3e01134640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.178+0000 7f6048f85640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60440ae020 0x7f6044135620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7feed99ec640 1 --2- 172.21.15.170:0/3866374398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feecc035f60 0x7feecc0c81e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feed4061be0 tx=0x7feed40a58c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75523052787defef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f6048f85640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60440ae020 0x7f6044135620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60468/0 (socket says 172.21.15.170:60468) 2023-10-30T17:32:38.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f6048f85640 1 -- 172.21.15.170:0/1045790948 learned_addr learned my addr 172.21.15.170:0/1045790948 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f6048f85640 10 monclient(hunting): get_auth_request con 0x7f60440ae020 auth_method 0 2023-10-30T17:32:38.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f6048f85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f6048f85640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f6048f85640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f3e01134640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2023-10-30T17:32:38.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f3e01134640 1 -- 172.21.15.170:0/2081714406 >> v2:172.21.15.170:3300/0 conn(0x7f3dfc0b7c90 msgr2=0x7f3dfc0a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f3e01134640 1 --2- 172.21.15.170:0/2081714406 >> v2:172.21.15.170:3300/0 conn(0x7f3dfc0b7c90 0x7f3dfc0a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f3e01134640 1 -- 172.21.15.170:0/2081714406 >> v1:172.21.15.177:6789/0 conn(0x7f3dfc13eb60 legacy=0x7f3dfc13efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f3e01134640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f3e01134640 1 monclient: found mon.noname-b 2023-10-30T17:32:38.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f3e01134640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f3e01134640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.179+0000 7f3e01134640 1 -- 172.21.15.170:0/2081714406 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc0aa9b0 con 0x7f3dfc144f50 2023-10-30T17:32:38.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.180+0000 7f6048f85640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.180+0000 7f6048f85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.180+0000 7f6048f85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.181+0000 7f9524cc9640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.181+0000 7f9524cc9640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.181+0000 7f9524cc9640 10 monclient: monmap: 2023-10-30T17:32:38.286 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.286 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.286 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.182305+0000 2023-10-30T17:32:38.286 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.182305+0000 2023-10-30T17:32:38.287 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.287 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.287 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.287 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.287 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.288 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.288 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.288 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 1 Processor -- start 2023-10-30T17:32:38.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 1 -- start start 2023-10-30T17:32:38.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 10 monclient: init 2023-10-30T17:32:38.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f3e01134640 1 --2- 172.21.15.170:0/2081714406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dfc144f50 0x7f3dfc145400 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009d90 tx=0x7f3df402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8dea0f69cbc5b58 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-10-30T17:32:38.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f95200b7c90 0x7f95200a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f95200b7c90 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:38.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f952013eb60 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:38.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9520144f50 0x7f9520145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 10 monclient(hunting): picked mon.noname-b con 0x7f9520144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95200aa740 con 0x7f952013eb60 2023-10-30T17:32:38.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95200a9500 con 0x7f95200b7c90 2023-10-30T17:32:38.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952013f6c0 con 0x7f9520144f50 2023-10-30T17:32:38.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.182+0000 7f9524cc9640 10 monclient(hunting): authenticate will time out at 1340.505990s 2023-10-30T17:32:38.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.183+0000 7f6048f85640 10 monclient(hunting): handle_auth_done global_id 4297 payload 824 2023-10-30T17:32:38.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.183+0000 7f6048f85640 1 -- 172.21.15.170:0/1045790948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6044070e00 msgr2=0x7f60441da4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.183+0000 7f6048f85640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6044070e00 0x7f60441da4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.183+0000 7f6048f85640 1 -- 172.21.15.170:0/1045790948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f604413daf0 msgr2=0x7f60440707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.183+0000 7f6048f85640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f604413daf0 0x7f60440707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.183+0000 7f6048f85640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.183+0000 7f6048f85640 1 monclient: found mon.noname-b 2023-10-30T17:32:38.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.183+0000 7f6048f85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.183+0000 7f6048f85640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.183+0000 7f6048f85640 1 -- 172.21.15.170:0/1045790948 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60340094d0 con 0x7f60440ae020 2023-10-30T17:32:38.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 1 -- 172.21.15.170:0/829057584 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0402ecd0 con 0x7f3a080ae020 2023-10-30T17:32:38.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:38.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 10 monclient: dump: 2023-10-30T17:32:38.296 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.297 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.297 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.297 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.297 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.297 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.297 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.298 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.298 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.298 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f9df259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dec035f60 0x7f9dec02e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f9df259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dec035f60 0x7f9dec02e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60474/0 (socket says 172.21.15.170:60474) 2023-10-30T17:32:38.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f9df259c640 1 -- 172.21.15.170:0/929183729 learned_addr learned my addr 172.21.15.170:0/929183729 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f9df259c640 10 monclient(hunting): get_auth_request con 0x7f9dec035f60 auth_method 0 2023-10-30T17:32:38.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f9df259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f9df259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f9df259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 1 -- 172.21.15.170:0/829057584 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a04032070 con 0x7f3a080ae020 2023-10-30T17:32:38.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f3a0e475640 5 monclient: authenticate success, global_id 4285 2023-10-30T17:32:38.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f3a0e475640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:38.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 1 -- 172.21.15.170:0/829057584 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04038620 con 0x7f3a080ae020 2023-10-30T17:32:38.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f3a0e475640 4 set_mon_vals no callback set 2023-10-30T17:32:38.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:38.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.185+0000 7f39fffff640 10 monclient: dump: 2023-10-30T17:32:38.303 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.303 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.303 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.303 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.303 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.304 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.304 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.304 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.304 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.304 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f3a0e475640 10 monclient: shutdown 2023-10-30T17:32:38.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f3a0e475640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f3a0e475640 1 -- 172.21.15.170:0/829057584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a080ae020 msgr2=0x7f3a080ac480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f3a0e475640 1 --2- 172.21.15.170:0/829057584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a080ae020 0x7f3a080ac480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3a0806ae10 tx=0x7f3a0402df20 comp rx=0 tx=0).stop 2023-10-30T17:32:38.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f3a0e475640 1 -- 172.21.15.170:0/829057584 shutdown_connections 2023-10-30T17:32:38.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f3a0e475640 1 --2- 172.21.15.170:0/829057584 >> v2:172.21.15.170:3300/0 conn(0x7f3a080ae400 0x7f3a0813ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f3a0e475640 1 --2- 172.21.15.170:0/829057584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a080ae020 0x7f3a080ac480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f3a0e475640 1 -- 172.21.15.170:0/829057584 >> 172.21.15.170:0/829057584 conn(0x7f3a080afbe0 msgr2=0x7f3a080afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f9df1d9b640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dec039510 0x7f9dec02fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f6048f85640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60440ae020 0x7f6044135620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f603800a7a0 tx=0x7f603800ac60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f60417fa640 10 client.?.objecter ms_handle_connect 0x7f60440ae020 2023-10-30T17:32:38.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f60417fa640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:38.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f3a0e475640 1 -- 172.21.15.170:0/829057584 shutdown_connections 2023-10-30T17:32:38.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.186+0000 7f3a0e475640 1 -- 172.21.15.170:0/829057584 wait complete. 2023-10-30T17:32:38.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f9df1d9b640 10 monclient(hunting): get_auth_request con 0x7f9dec039510 auth_method 0 2023-10-30T17:32:38.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f9df1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f9df1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f9df1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f3a0e475640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f3a0e475640 10 monclient: monmap: 2023-10-30T17:32:38.310 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.310 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:38.311 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:38.311 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:38.311 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.311 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.311 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:38.311 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:38.312 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:38.312 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f2c70cd0640 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2023-10-30T17:32:38.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f2c70cd0640 1 -- 172.21.15.170:0/504639037 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c6c0a8c00 msgr2=0x7f2c6c0a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f2c70cd0640 1 --2- 172.21.15.170:0/504639037 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c6c0a8c00 0x7f2c6c0a5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f2c70cd0640 1 -- 172.21.15.170:0/504639037 >> v1:172.21.15.177:6789/0 conn(0x7f2c6c0b0330 legacy=0x7f2c6c0a6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f2c70cd0640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f2c70cd0640 1 monclient: found mon.noname-c 2023-10-30T17:32:38.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.187+0000 7f2c70cd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f2c70cd0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f2c70cd0640 1 -- 172.21.15.170:0/504639037 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c0a8500 con 0x7f2c6c0a7220 2023-10-30T17:32:38.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f9df1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f9df1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f9df1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f3a0e475640 1 librados: starting msgr at 2023-10-30T17:32:38.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f3a0e475640 1 librados: starting objecter 2023-10-30T17:32:38.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f3a0e475640 1 Processor -- start 2023-10-30T17:32:38.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f3a0e475640 1 -- start start 2023-10-30T17:32:38.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f3a0e475640 1 librados: setting wanted keys 2023-10-30T17:32:38.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f3a0e475640 1 librados: calling monclient init 2023-10-30T17:32:38.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.188+0000 7f3a0e475640 10 monclient: init 2023-10-30T17:32:38.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:32:38.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a080ae020 0x7f3a08138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 10 monclient(hunting): picked mon.noname-a con 0x7f3a080ae020 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:38.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a0813daf0 0x7f3a081393e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 10 monclient(hunting): picked mon.noname-b con 0x7f3a0813daf0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:38.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a0811f370 0x7f3a081da3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 10 monclient(hunting): picked mon.noname-c con 0x7f3a0811f370 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08071660 con 0x7f3a080ae020 2023-10-30T17:32:38.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08071d10 con 0x7f3a0813daf0 2023-10-30T17:32:38.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08072130 con 0x7f3a0811f370 2023-10-30T17:32:38.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.189+0000 7f3a0e475640 10 monclient(hunting): authenticate will time out at 1340.512552s 2023-10-30T17:32:38.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.191+0000 7f9df259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.191+0000 7f9df259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.191+0000 7f9df259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f9df1d9b640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2023-10-30T17:32:38.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f9df1d9b640 1 -- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dec035f60 msgr2=0x7f9dec02e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f9df1d9b640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dec035f60 0x7f9dec02e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f9df1d9b640 1 -- 172.21.15.170:0/929183729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec0bbe00 msgr2=0x7f9dec0301d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f9df1d9b640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec0bbe00 0x7f9dec0301d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f9df1d9b640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f9df1d9b640 1 monclient: found mon.noname-c 2023-10-30T17:32:38.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f9df1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f9df1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f9df1d9b640 1 -- 172.21.15.170:0/929183729 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4071040 con 0x7f9dec039510 2023-10-30T17:32:38.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.192+0000 7f951e59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f95200b7c90 0x7f95200a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.196+0000 7f951e59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f95200b7c90 0x7f95200a8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60490/0 (socket says 172.21.15.170:60490) 2023-10-30T17:32:38.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.196+0000 7f951e59c640 1 -- 172.21.15.170:0/1241126293 learned_addr learned my addr 172.21.15.170:0/1241126293 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.196+0000 7f951e59c640 10 monclient(hunting): get_auth_request con 0x7f95200b7c90 auth_method 0 2023-10-30T17:32:38.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.196+0000 7f951e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.196+0000 7f951e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.196+0000 7f951e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.197+0000 7f9df1d9b640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dec039510 0x7f9dec02fc90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9df4069460 tx=0x7f9df4061df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.197+0000 7f9de2ffd640 10 client.?.objecter ms_handle_connect 0x7f9dec039510 2023-10-30T17:32:38.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.197+0000 7f9de2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:38.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.198+0000 7f951ed9d640 1 --2- 172.21.15.170:0/1241126293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9520144f50 0x7f9520145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 1 -- 172.21.15.170:0/2081714406 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df402cbc0 con 0x7f3dfc144f50 2023-10-30T17:32:38.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:38.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 10 monclient: dump: 2023-10-30T17:32:38.329 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.329 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.329 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.331 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.331 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 1 -- 172.21.15.170:0/2081714406 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df402cd20 con 0x7f3dfc144f50 2023-10-30T17:32:38.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 5 monclient: authenticate success, global_id 4265 2023-10-30T17:32:38.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:38.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:38.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 4 set_mon_vals no callback set 2023-10-30T17:32:38.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 1 -- 172.21.15.170:0/2081714406 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4034660 con 0x7f3dfc144f50 2023-10-30T17:32:38.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:38.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3dfb7fe640 10 monclient: dump: 2023-10-30T17:32:38.334 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.334 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.334 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.334 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.334 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.335 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.335 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.335 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.335 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.335 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 10 monclient: shutdown 2023-10-30T17:32:38.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 1 -- 172.21.15.170:0/2081714406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dfc144f50 msgr2=0x7f3dfc145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 1 --2- 172.21.15.170:0/2081714406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dfc144f50 0x7f3dfc145400 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009d90 tx=0x7f3df402be10 comp rx=0 tx=0).stop 2023-10-30T17:32:38.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 1 -- 172.21.15.170:0/2081714406 shutdown_connections 2023-10-30T17:32:38.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 1 --2- 172.21.15.170:0/2081714406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dfc144f50 0x7f3dfc145400 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 1 --2- 172.21.15.170:0/2081714406 >> v2:172.21.15.170:3300/0 conn(0x7f3dfc0b7c90 0x7f3dfc0a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 1 -- 172.21.15.170:0/2081714406 >> 172.21.15.170:0/2081714406 conn(0x7f3dfc071da0 msgr2=0x7f3dfc0b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 1 -- 172.21.15.170:0/2081714406 shutdown_connections 2023-10-30T17:32:38.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.200+0000 7f3e02b97640 1 -- 172.21.15.170:0/2081714406 wait complete. 2023-10-30T17:32:38.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.201+0000 7f3e02b97640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.201+0000 7f3e02b97640 10 monclient: monmap: 2023-10-30T17:32:38.337 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.338 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:38.338 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:38.338 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:38.338 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.338 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.339 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:38.339 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:38.339 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:38.339 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.202+0000 7f951e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.340 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.202+0000 7f951e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.340 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.202+0000 7f3e02b97640 1 librados: starting msgr at 2023-10-30T17:32:38.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.202+0000 7f951e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.202+0000 7f3e02b97640 1 librados: starting objecter 2023-10-30T17:32:38.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.202+0000 7f3e02b97640 1 Processor -- start 2023-10-30T17:32:38.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.202+0000 7f3e02b97640 1 -- start start 2023-10-30T17:32:38.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.202+0000 7f3e02b97640 1 librados: setting wanted keys 2023-10-30T17:32:38.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.202+0000 7f3e02b97640 1 librados: calling monclient init 2023-10-30T17:32:38.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.202+0000 7f3e02b97640 10 monclient: init 2023-10-30T17:32:38.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 1 -- 172.21.15.170:0/929183729 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df40b4430 con 0x7f9dec039510 2023-10-30T17:32:38.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:38.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 10 monclient: dump: 2023-10-30T17:32:38.344 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.344 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.345 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.345 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.345 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.345 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.345 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.345 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.346 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.346 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9df8e4b640 5 monclient: authenticate success, global_id 4268 2023-10-30T17:32:38.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 1 -- 172.21.15.170:0/929183729 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df40b4590 con 0x7f9dec039510 2023-10-30T17:32:38.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 1 -- 172.21.15.170:0/929183729 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df40b4900 con 0x7f9dec039510 2023-10-30T17:32:38.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.348 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:38.348 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9de2ffd640 10 monclient: dump: 2023-10-30T17:32:38.348 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.348 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.348 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.349 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.349 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.349 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.349 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.349 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.350 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.350 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9df8e4b640 10 monclient: _renew_subs 2023-10-30T17:32:38.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9df0d99640 4 set_mon_vals no callback set 2023-10-30T17:32:38.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9df8e4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dec02eb80 con 0x7f9dec039510 2023-10-30T17:32:38.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9df8e4b640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9df8e4b640 10 monclient: _renew_subs 2023-10-30T17:32:38.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9df8e4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dec030710 con 0x7f9dec039510 2023-10-30T17:32:38.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f9df8e4b640 1 librados: init done 2023-10-30T17:32:38.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:32:38.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3dfc0b7c90 0x7f3dfc144450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 10 monclient(hunting): picked mon.noname-a con 0x7f3dfc0b7c90 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:38.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dfc13eb60 0x7f3dfc0ab360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 10 monclient(hunting): picked mon.noname-b con 0x7f3dfc13eb60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:38.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dfc144f50 0x7f3dfc1d6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.354 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f951ed9d640 10 monclient(hunting): get_auth_request con 0x7f9520144f50 auth_method 0 2023-10-30T17:32:38.354 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f951ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.354 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feecbfff640 1 -- 172.21.15.170:0/3866374398 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed40ab8e0 con 0x7feecc035f60 2023-10-30T17:32:38.354 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feecbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.355 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 10 monclient(hunting): picked mon.noname-c con 0x7f3dfc144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.355 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.355 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0ab8a0 con 0x7f3dfc0b7c90 2023-10-30T17:32:38.356 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.356 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0ac5f0 con 0x7f3dfc13eb60 2023-10-30T17:32:38.356 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.356 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0ac860 con 0x7f3dfc144f50 2023-10-30T17:32:38.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f3e02b97640 10 monclient(hunting): authenticate will time out at 1340.526514s 2023-10-30T17:32:38.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feecbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:38.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feecbfff640 10 monclient: dump: 2023-10-30T17:32:38.358 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.359 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.359 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.359 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.359 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.359 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.360 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.360 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.360 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.360 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feecbfff640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feecbfff640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feecbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feecbfff640 1 -- 172.21.15.170:0/3866374398 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed40a7040 con 0x7feecc035f60 2023-10-30T17:32:38.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feeda1ed640 5 monclient: authenticate success, global_id 4291 2023-10-30T17:32:38.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feeda1ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:38.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feecbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feecbfff640 1 -- 172.21.15.170:0/3866374398 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed40abd40 con 0x7feecc035f60 2023-10-30T17:32:38.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feeda1ed640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:38.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7feeda1ed640 4 set_mon_vals no callback set 2023-10-30T17:32:38.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f951ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.203+0000 7f951ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.364 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feecbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.364 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feecbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:38.364 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feecbfff640 10 monclient: dump: 2023-10-30T17:32:38.364 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.364 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.365 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.365 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.365 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.365 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.365 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.366 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.366 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.366 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.366 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7f9df8e4b640 10 client.4268.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:38.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7f9df8e4b640 15 client.4268.objecter _session_command_op_assign -1 1 2023-10-30T17:32:38.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7f9df8e4b640 20 client.4268.objecter _get_session osd=-1 returning homeless 2023-10-30T17:32:38.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7f9df8e4b640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7f9df8e4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9dec0c88f0 con 0x7f9dec039510 2023-10-30T17:32:38.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feeda1ed640 10 monclient: shutdown 2023-10-30T17:32:38.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feeda1ed640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feeda1ed640 1 -- 172.21.15.170:0/3866374398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feecc035f60 msgr2=0x7feecc0c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feeda1ed640 1 --2- 172.21.15.170:0/3866374398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feecc035f60 0x7feecc0c81e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feed4061be0 tx=0x7feed40a58c0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feeda1ed640 1 -- 172.21.15.170:0/3866374398 shutdown_connections 2023-10-30T17:32:38.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feeda1ed640 1 --2- 172.21.15.170:0/3866374398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feecc035f60 0x7feecc0c81e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feeda1ed640 1 --2- 172.21.15.170:0/3866374398 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feecc0398f0 0x7feecc035a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feeda1ed640 1 -- 172.21.15.170:0/3866374398 >> 172.21.15.170:0/3866374398 conn(0x7feecc0070a0 msgr2=0x7feecc037df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feeda1ed640 1 -- 172.21.15.170:0/3866374398 shutdown_connections 2023-10-30T17:32:38.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7feeda1ed640 1 -- 172.21.15.170:0/3866374398 wait complete. 2023-10-30T17:32:38.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7f9494b1f640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f9490070100 0x7f9490140010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:39550/0 (socket says 172.21.15.170:39550) 2023-10-30T17:32:38.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.204+0000 7f9494b1f640 1 -- 172.21.15.170:0/2402540863 learned_addr learned my addr 172.21.15.170:0/2402540863 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:38.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7f951ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7feeda1ed640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7feeda1ed640 10 monclient: monmap: 2023-10-30T17:32:38.373 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.373 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:38.373 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:38.373 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:38.373 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.373 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.374 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:38.374 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:38.374 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:38.374 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7f951ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7f951ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7feeda1ed640 1 librados: starting msgr at 2023-10-30T17:32:38.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7feeda1ed640 1 librados: starting objecter 2023-10-30T17:32:38.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7feeda1ed640 1 Processor -- start 2023-10-30T17:32:38.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7feeda1ed640 1 -- start start 2023-10-30T17:32:38.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7feeda1ed640 1 librados: setting wanted keys 2023-10-30T17:32:38.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7feeda1ed640 1 librados: calling monclient init 2023-10-30T17:32:38.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7feeda1ed640 10 monclient: init 2023-10-30T17:32:38.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7f951e59c640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2023-10-30T17:32:38.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7f951e59c640 1 -- 172.21.15.170:0/1241126293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9520144f50 msgr2=0x7f9520145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.205+0000 7f951e59c640 1 --2- 172.21.15.170:0/1241126293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9520144f50 0x7f9520145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7f951e59c640 1 -- 172.21.15.170:0/1241126293 >> v1:172.21.15.170:6789/0 conn(0x7f952013eb60 legacy=0x7f952013efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7f951e59c640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7f951e59c640 1 monclient: found mon.noname-c 2023-10-30T17:32:38.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7f951e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7f951e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7f951e59c640 1 -- 172.21.15.170:0/1241126293 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95200aa9b0 con 0x7f95200b7c90 2023-10-30T17:32:38.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7f3a0dc74640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a0811f370 0x7f3a081da3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:32:38.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feecc035f60 0x7feecc0c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 10 monclient(hunting): picked mon.noname-b con 0x7feecc035f60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:38.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feecc039510 0x7feecc0cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 10 monclient(hunting): picked mon.noname-a con 0x7feecc039510 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:38.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feecc0ca300 0x7feecc0bd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 10 monclient(hunting): picked mon.noname-c con 0x7feecc0ca300 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feecc031430 con 0x7feecc039510 2023-10-30T17:32:38.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feecc031640 con 0x7feecc035f60 2023-10-30T17:32:38.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feecc0bdb30 con 0x7feecc0ca300 2023-10-30T17:32:38.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.206+0000 7feeda1ed640 10 monclient(hunting): authenticate will time out at 1340.529709s 2023-10-30T17:32:38.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.209+0000 7f951e59c640 1 --2- 172.21.15.170:0/1241126293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95200b7c90 0x7f95200a8b80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9510009870 tx=0x7f951002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbfc4c83f218782c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.209+0000 7f3a0dc74640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a0811f370 0x7f3a081da3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:53274/0 (socket says 172.21.15.170:53274) 2023-10-30T17:32:38.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.209+0000 7f3a0dc74640 1 -- 172.21.15.170:0/784572801 learned_addr learned my addr 172.21.15.170:0/784572801 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.209+0000 7f3a0dc74640 10 monclient(hunting): get_auth_request con 0x7f3a0811f370 auth_method 0 2023-10-30T17:32:38.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.209+0000 7f3a0dc74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.209+0000 7f3a0dc74640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.209+0000 7f3a0dc74640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.214+0000 7f3a0cc72640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a0813daf0 0x7f3a081393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.214+0000 7f3dfbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dfc13eb60 0x7f3dfc0ab360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.214+0000 7f3a0cc72640 10 monclient(hunting): get_auth_request con 0x7f3a0813daf0 auth_method 0 2023-10-30T17:32:38.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.214+0000 7f3a0cc72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.214+0000 7f3a0cc72640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.214+0000 7f3a0cc72640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.215+0000 7f3a0cc72640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.215+0000 7f3a0cc72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.215+0000 7f3a0cc72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.215+0000 7f3dfbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dfc13eb60 0x7f3dfc0ab360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60510/0 (socket says 172.21.15.170:60510) 2023-10-30T17:32:38.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.215+0000 7f3dfbfff640 1 -- 172.21.15.170:0/1997246290 learned_addr learned my addr 172.21.15.170:0/1997246290 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.215+0000 7f3dfbfff640 10 monclient(hunting): get_auth_request con 0x7f3dfc13eb60 auth_method 0 2023-10-30T17:32:38.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.215+0000 7f3dfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.216+0000 7f3dfbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.216+0000 7f3dfbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.217+0000 7f3a0dc74640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.217+0000 7f3a0dc74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.217+0000 7f3a0dc74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3a0cc72640 10 monclient(hunting): handle_auth_done global_id 4306 payload 824 2023-10-30T17:32:38.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3a0cc72640 1 -- 172.21.15.170:0/784572801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a0811f370 msgr2=0x7f3a081da3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3a0cc72640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a0811f370 0x7f3a081da3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3a0cc72640 1 -- 172.21.15.170:0/784572801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a080ae020 msgr2=0x7f3a08138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3a0cc72640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a080ae020 0x7f3a08138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3a0cc72640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3a0cc72640 1 monclient: found mon.noname-b 2023-10-30T17:32:38.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3a0cc72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.393 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3a0cc72640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.393 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3a0cc72640 1 -- 172.21.15.170:0/784572801 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a04008550 con 0x7f3a0813daf0 2023-10-30T17:32:38.393 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.218+0000 7f3e01134640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dfc144f50 0x7f3dfc1d6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 1 -- 172.21.15.170:0/1045790948 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603800f040 con 0x7f60440ae020 2023-10-30T17:32:38.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:38.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 10 monclient: dump: 2023-10-30T17:32:38.396 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.396 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.396 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.396 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.397 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.397 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.397 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.397 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.397 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.398 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 1 -- 172.21.15.170:0/1045790948 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603800d8a0 con 0x7f60440ae020 2023-10-30T17:32:38.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 1 -- 172.21.15.170:0/1045790948 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603800dc30 con 0x7f60440ae020 2023-10-30T17:32:38.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:38.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f60417fa640 10 monclient: dump: 2023-10-30T17:32:38.400 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.400 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.400 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.400 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.400 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.401 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.401 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.401 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.401 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.401 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.401 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f6049f87640 5 monclient: authenticate success, global_id 4297 2023-10-30T17:32:38.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f6049f87640 10 monclient: _renew_subs 2023-10-30T17:32:38.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f6049f87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f6042ffd640 4 set_mon_vals no callback set 2023-10-30T17:32:38.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6044070070 con 0x7f60440ae020 2023-10-30T17:32:38.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f6049f87640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.403 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f6049f87640 10 monclient: _renew_subs 2023-10-30T17:32:38.403 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f6049f87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.403 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60441dac00 con 0x7f60440ae020 2023-10-30T17:32:38.403 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.221+0000 7f6049f87640 1 librados: init done 2023-10-30T17:32:38.404 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 1 -- 172.21.15.170:0/1241126293 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95100317c0 con 0x7f95200b7c90 2023-10-30T17:32:38.404 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f6049f87640 10 client.4297.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:38.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f6049f87640 15 client.4297.objecter _session_command_op_assign -1 1 2023-10-30T17:32:38.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f6049f87640 20 client.4297.objecter _get_session osd=-1 returning homeless 2023-10-30T17:32:38.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f6049f87640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f6049f87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60440ad6c0 con 0x7f60440ae020 2023-10-30T17:32:38.406 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f2c70cd0640 1 --2- 172.21.15.170:0/504639037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6c0a7220 0x7f2c6c13cad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009d90 tx=0x7f2c6402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc0cba807f210c70 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:38.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 10 monclient: dump: 2023-10-30T17:32:38.408 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.408 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.408 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.408 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.410 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.410 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.410 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.410 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.410 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 1 -- 172.21.15.170:0/1241126293 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9510031920 con 0x7f95200b7c90 2023-10-30T17:32:38.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 1 -- 172.21.15.170:0/1241126293 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510031cb0 con 0x7f95200b7c90 2023-10-30T17:32:38.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:38.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f951dd9b640 10 monclient: dump: 2023-10-30T17:32:38.412 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.412 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.412 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.412 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.412 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.412 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.413 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.413 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.413 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.413 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f9524cc9640 5 monclient: authenticate success, global_id 4303 2023-10-30T17:32:38.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f9524cc9640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:38.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.222+0000 7f9524cc9640 4 set_mon_vals no callback set 2023-10-30T17:32:38.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f3dfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f3dfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f3dfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.416 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f9524cc9640 10 monclient: shutdown 2023-10-30T17:32:38.416 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f9524cc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.416 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f9524cc9640 1 -- 172.21.15.170:0/1241126293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95200b7c90 msgr2=0x7f95200a8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.417 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f9524cc9640 1 --2- 172.21.15.170:0/1241126293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95200b7c90 0x7f95200a8b80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9510009870 tx=0x7f951002b860 comp rx=0 tx=0).stop 2023-10-30T17:32:38.417 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f9524cc9640 1 -- 172.21.15.170:0/1241126293 shutdown_connections 2023-10-30T17:32:38.417 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f9524cc9640 1 --2- 172.21.15.170:0/1241126293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9520144f50 0x7f9520145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.417 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f9524cc9640 1 --2- 172.21.15.170:0/1241126293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95200b7c90 0x7f95200a8b80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.417 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f9524cc9640 1 -- 172.21.15.170:0/1241126293 >> 172.21.15.170:0/1241126293 conn(0x7f9520071da0 msgr2=0x7f95200b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.418 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f9524cc9640 1 -- 172.21.15.170:0/1241126293 shutdown_connections 2023-10-30T17:32:38.418 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.223+0000 7f9524cc9640 1 -- 172.21.15.170:0/1241126293 wait complete. 2023-10-30T17:32:38.418 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.224+0000 7f3a0cc72640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a0813daf0 0x7f3a081393e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3a00007e60 tx=0x7f3a0000e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.418 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.224+0000 7f39fdffb640 10 client.?.objecter ms_handle_connect 0x7f3a0813daf0 2023-10-30T17:32:38.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.224+0000 7f39fdffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:38.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.224+0000 7f9524cc9640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.224+0000 7f9524cc9640 10 monclient: monmap: 2023-10-30T17:32:38.420 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.420 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:38.420 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:38.420 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:38.420 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.420 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.421 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:38.421 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:38.421 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:38.421 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.224+0000 7feed91eb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feecc035f60 0x7feecc0c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7feed91eb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feecc035f60 0x7feecc0c6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60512/0 (socket says 172.21.15.170:60512) 2023-10-30T17:32:38.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7feed91eb640 1 -- 172.21.15.170:0/3136618199 learned_addr learned my addr 172.21.15.170:0/3136618199 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7feed91eb640 10 monclient(hunting): get_auth_request con 0x7feecc035f60 auth_method 0 2023-10-30T17:32:38.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7feed91eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7feed91eb640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7feed91eb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 1 librados: starting msgr at 2023-10-30T17:32:38.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 1 librados: starting objecter 2023-10-30T17:32:38.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 1 Processor -- start 2023-10-30T17:32:38.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 1 -- start start 2023-10-30T17:32:38.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 1 librados: setting wanted keys 2023-10-30T17:32:38.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 1 librados: calling monclient init 2023-10-30T17:32:38.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 10 monclient: init 2023-10-30T17:32:38.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:32:38.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95200b7c90 0x7f9520142450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 10 monclient(hunting): picked mon.noname-b con 0x7f95200b7c90 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:38.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.225+0000 7f9524cc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f952013eb60 0x7f95201431e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 10 monclient(hunting): picked mon.noname-a con 0x7f952013eb60 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:38.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9520144f50 0x7f95201d6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f9520144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9520143bf0 con 0x7f952013eb60 2023-10-30T17:32:38.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9520143720 con 0x7f95200b7c90 2023-10-30T17:32:38.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9520144840 con 0x7f9520144f50 2023-10-30T17:32:38.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.226+0000 7f9524cc9640 10 monclient(hunting): authenticate will time out at 1340.549320s 2023-10-30T17:32:38.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.227+0000 7feed91eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.227+0000 7feed91eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.227+0000 7feed91eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.231+0000 7f3dfbfff640 10 monclient(hunting): handle_auth_done global_id 4309 payload 824 2023-10-30T17:32:38.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.231+0000 7f3dfbfff640 1 -- 172.21.15.170:0/1997246290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dfc144f50 msgr2=0x7f3dfc1d6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.231+0000 7f3dfbfff640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dfc144f50 0x7f3dfc1d6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.231+0000 7f3dfbfff640 1 -- 172.21.15.170:0/1997246290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3dfc0b7c90 msgr2=0x7f3dfc144450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.231+0000 7f3dfbfff640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3dfc0b7c90 0x7f3dfc144450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.231+0000 7f3dfbfff640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.231+0000 7f3dfbfff640 1 monclient: found mon.noname-b 2023-10-30T17:32:38.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.231+0000 7f3dfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.231+0000 7f3dfbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.231+0000 7f3dfbfff640 1 -- 172.21.15.170:0/1997246290 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df4009a70 con 0x7f3dfc13eb60 2023-10-30T17:32:38.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.234+0000 7f9de2ffd640 1 -- 172.21.15.170:0/929183729 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9df40b4a60 con 0x7f9dec039510 2023-10-30T17:32:38.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.234+0000 7f9de2ffd640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9dd406c800 0x7f9dd406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.235+0000 7f9df259c640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9dd406c800 0x7f9dd406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.235+0000 7f9df259c640 10 monclient: get_auth_request con 0x7f9dd406c800 auth_method 0 2023-10-30T17:32:38.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.235+0000 7f9df259c640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9dd406c800 0x7f9dd406ecc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9de4006fd0 tx=0x7f9de4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.435 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.235+0000 7f9de2ffd640 10 client.4268.objecter ms_handle_connect 0x7f9dd406c800 2023-10-30T17:32:38.435 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 1 -- 172.21.15.170:0/929183729 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9df40f5b50 con 0x7f9dec039510 2023-10-30T17:32:38.435 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 10 client.4268.objecter ms_dispatch 0x7f9df4000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:38.435 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 3 client.4268.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:38.435 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 3 client.4268.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:38.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 10 client.4268.objecter checking command 1 2023-10-30T17:32:38.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9495320640 1 --2- 172.21.15.170:0/2402540863 >> v2:172.21.15.170:3300/0 conn(0x7f949013d040 0x7f94900afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] conn(0x7f9dd4072be0 0x7f9dd4075000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 20 client.4268.objecter _get_session s=0x7f9dd4072510 osd=4 3 2023-10-30T17:32:38.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 20 client.4268.objecter put_session s=0x7f9dd4072510 osd=4 3 2023-10-30T17:32:38.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 15 client.4268.objecter _session_command_op_remove -1 1 2023-10-30T17:32:38.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 20 client.4268.objecter _get_session s=0x7f9dd4072510 osd=4 3 2023-10-30T17:32:38.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 20 client.4268.objecter get_session s=0x7f9dd4072510 osd=4 3 2023-10-30T17:32:38.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 15 client.4268.objecter _session_command_op_assign 4 1 2023-10-30T17:32:38.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 20 client.4268.objecter put_session s=0x7f9dd4072510 osd=4 4 2023-10-30T17:32:38.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 10 client.4268.objecter _send_command 1 2023-10-30T17:32:38.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 1 -- 172.21.15.170:0/929183729 --> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9dd40756b0 con 0x7f9dd4072be0 2023-10-30T17:32:38.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.238+0000 7f9de2ffd640 20 client.4268.objecter dump_active .. 0 homeless 2023-10-30T17:32:38.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.239+0000 7f9df2d9d640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] conn(0x7f9dd4072be0 0x7f9dd4075000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9517fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f952013eb60 0x7f95201431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9df2d9d640 10 monclient: get_auth_request con 0x7f9dd4072be0 auth_method 0 2023-10-30T17:32:38.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9495320640 10 monclient(hunting): get_auth_request con 0x7f949013d040 auth_method 0 2023-10-30T17:32:38.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9495320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9495320640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9495320640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9517fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f952013eb60 0x7f95201431e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:39866/0 (socket says 172.21.15.170:39866) 2023-10-30T17:32:38.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9517fff640 1 -- 172.21.15.170:0/1993976595 learned_addr learned my addr 172.21.15.170:0/1993976595 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9517fff640 10 monclient(hunting): get_auth_request con 0x7f952013eb60 auth_method 0 2023-10-30T17:32:38.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9517fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9517fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.240+0000 7f9517fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.241+0000 7f9495320640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.241+0000 7f9495320640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.241+0000 7f9495320640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.242+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.242+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.445 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.242+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.445 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.243+0000 7feed89ea640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feecc039510 0x7feecc0cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.445 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.243+0000 7f9df2d9d640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] conn(0x7f9dd4072be0 0x7f9dd4075000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.446 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.243+0000 7f9de2ffd640 10 client.4268.objecter ms_handle_connect 0x7f9dd4072be0 2023-10-30T17:32:38.446 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.246+0000 7f9517fff640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2023-10-30T17:32:38.446 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.246+0000 7f9517fff640 1 -- 172.21.15.170:0/1993976595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9520144f50 msgr2=0x7f95201d6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.246+0000 7f9517fff640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9520144f50 0x7f95201d6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.246+0000 7f9517fff640 1 -- 172.21.15.170:0/1993976595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95200b7c90 msgr2=0x7f9520142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.246+0000 7f9517fff640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95200b7c90 0x7f9520142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.246+0000 7f9517fff640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.246+0000 7f9517fff640 1 monclient: found mon.noname-a 2023-10-30T17:32:38.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.246+0000 7f9517fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.246+0000 7f9517fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:38.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.246+0000 7f9517fff640 1 -- 172.21.15.170:0/1993976595 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95100094d0 con 0x7f952013eb60 2023-10-30T17:32:38.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.247+0000 7f9495320640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2023-10-30T17:32:38.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.247+0000 7f9495320640 1 -- 172.21.15.170:0/2402540863 >> v1:172.21.15.177:6789/0 conn(0x7f9490070100 legacy=0x7f9490140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.247+0000 7f9495320640 1 -- 172.21.15.170:0/2402540863 >> v1:172.21.15.170:6789/0 conn(0x7f94900704e0 legacy=0x7f949013c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.247+0000 7f9495320640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.247+0000 7f9495320640 1 monclient: found mon.noname-a 2023-10-30T17:32:38.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.247+0000 7f9495320640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.247+0000 7f9495320640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:38.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.247+0000 7f9495320640 1 -- 172.21.15.170:0/2402540863 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949013f810 con 0x7f949013d040 2023-10-30T17:32:38.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 1 -- 172.21.15.170:0/504639037 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6402cbc0 con 0x7f2c6c0a7220 2023-10-30T17:32:38.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:38.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 10 monclient: dump: 2023-10-30T17:32:38.451 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.451 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.451 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.452 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.452 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.452 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.452 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.452 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.453 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.453 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 1 -- 172.21.15.170:0/504639037 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6402cd20 con 0x7f2c6c0a7220 2023-10-30T17:32:38.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 1 -- 172.21.15.170:0/504639037 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64034660 con 0x7f2c6c0a7220 2023-10-30T17:32:38.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:38.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c6affd640 10 monclient: dump: 2023-10-30T17:32:38.455 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.455 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.455 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.455 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.455 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.456 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.456 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.456 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.456 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.456 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7feed99ec640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feecc0ca300 0x7feecc0bd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c72733640 5 monclient: authenticate success, global_id 4294 2023-10-30T17:32:38.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c72733640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:38.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f2c72733640 4 set_mon_vals no callback set 2023-10-30T17:32:38.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f3dfbfff640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dfc13eb60 0x7f3dfc0ab360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3de800c4d0 tx=0x7f3de800c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f3df97fa640 10 client.?.objecter ms_handle_connect 0x7f3dfc13eb60 2023-10-30T17:32:38.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.248+0000 7f3df97fa640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:38.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7f2c72733640 10 monclient: shutdown 2023-10-30T17:32:38.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7f2c72733640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7f2c72733640 1 -- 172.21.15.170:0/504639037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6c0a7220 msgr2=0x7f2c6c13cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7f2c72733640 1 --2- 172.21.15.170:0/504639037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6c0a7220 0x7f2c6c13cad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009d90 tx=0x7f2c6402be10 comp rx=0 tx=0).stop 2023-10-30T17:32:38.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7f2c72733640 1 -- 172.21.15.170:0/504639037 shutdown_connections 2023-10-30T17:32:38.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7f2c72733640 1 --2- 172.21.15.170:0/504639037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6c0a7220 0x7f2c6c13cad0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7f2c72733640 1 --2- 172.21.15.170:0/504639037 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c6c0a8c00 0x7f2c6c0a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7f2c72733640 1 -- 172.21.15.170:0/504639037 >> 172.21.15.170:0/504639037 conn(0x7f2c6c0b3560 msgr2=0x7f2c6c0af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7f2c72733640 1 -- 172.21.15.170:0/504639037 shutdown_connections 2023-10-30T17:32:38.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7feed89ea640 10 monclient(hunting): get_auth_request con 0x7feecc039510 auth_method 0 2023-10-30T17:32:38.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7feed89ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7feed89ea640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7feed89ea640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.249+0000 7f2c72733640 1 -- 172.21.15.170:0/504639037 wait complete. 2023-10-30T17:32:38.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7f2c72733640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7f2c72733640 10 monclient: monmap: 2023-10-30T17:32:38.463 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.463 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:38.463 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:38.464 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:38.464 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.464 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.464 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:38.464 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:38.465 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:38.465 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7f2c72733640 1 librados: starting msgr at 2023-10-30T17:32:38.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7f2c72733640 1 librados: starting objecter 2023-10-30T17:32:38.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7f2c72733640 1 Processor -- start 2023-10-30T17:32:38.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7feed99ec640 10 monclient(hunting): get_auth_request con 0x7feecc0ca300 auth_method 0 2023-10-30T17:32:38.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7feed99ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7feed99ec640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7feed99ec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7f2c72733640 1 -- start start 2023-10-30T17:32:38.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7f2c72733640 1 librados: setting wanted keys 2023-10-30T17:32:38.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7f2c72733640 1 librados: calling monclient init 2023-10-30T17:32:38.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.250+0000 7f2c72733640 10 monclient: init 2023-10-30T17:32:38.468 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7feed99ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.468 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7feed99ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.468 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7feed99ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.469 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.469 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:32:38.469 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c6c0a7220 0x7f2c6c071900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.469 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 10 monclient(hunting): picked mon.noname-c con 0x7f2c6c0a7220 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6c0a8c00 0x7f2c6c12df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 10 monclient(hunting): picked mon.noname-b con 0x7f2c6c0a8c00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:38.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c6c0b0330 0x7f2c6c12c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 10 monclient(hunting): picked mon.noname-a con 0x7f2c6c0b0330 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:38.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c12bf00 con 0x7f2c6c0b0330 2023-10-30T17:32:38.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c071e40 con 0x7f2c6c0a8c00 2023-10-30T17:32:38.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c12a880 con 0x7f2c6c0a7220 2023-10-30T17:32:38.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.251+0000 7f2c72733640 10 monclient(hunting): authenticate will time out at 1340.574269s 2023-10-30T17:32:38.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 1 -- 172.21.15.170:0/784572801 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00003b90 con 0x7f3a0813daf0 2023-10-30T17:32:38.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:38.474 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 10 monclient: dump: 2023-10-30T17:32:38.474 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.474 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.474 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.474 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.475 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.475 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.475 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.475 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.475 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.476 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.476 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.476 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.476 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.476 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 1 -- 172.21.15.170:0/784572801 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a00003cf0 con 0x7f3a0813daf0 2023-10-30T17:32:38.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 1 -- 172.21.15.170:0/784572801 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00004270 con 0x7f3a0813daf0 2023-10-30T17:32:38.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:38.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39fdffb640 10 monclient: dump: 2023-10-30T17:32:38.478 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.478 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.478 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.478 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.478 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.479 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.479 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.479 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.479 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.479 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.479 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f3a0e475640 5 monclient: authenticate success, global_id 4306 2023-10-30T17:32:38.480 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f3a0e475640 10 monclient: _renew_subs 2023-10-30T17:32:38.480 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f3a0e475640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.480 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a08070990 con 0x7f3a0813daf0 2023-10-30T17:32:38.480 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f39ff7fe640 4 set_mon_vals no callback set 2023-10-30T17:32:38.480 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f3a0e475640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f3a0e475640 10 monclient: _renew_subs 2023-10-30T17:32:38.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f3a0e475640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0806ea00 con 0x7f3a0813daf0 2023-10-30T17:32:38.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.252+0000 7f3a0e475640 1 librados: init done 2023-10-30T17:32:38.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.253+0000 7f3a0e475640 10 client.4306.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:38.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.253+0000 7f3a0e475640 15 client.4306.objecter _session_command_op_assign -1 1 2023-10-30T17:32:38.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.253+0000 7f3a0e475640 20 client.4306.objecter _get_session osd=-1 returning homeless 2023-10-30T17:32:38.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.253+0000 7f3a0e475640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.253+0000 7f3a0e475640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.253+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a0813b2d0 con 0x7f3a0813daf0 2023-10-30T17:32:38.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.254+0000 7feed89ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.254+0000 7feed89ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.254+0000 7feed89ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7feed89ea640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2023-10-30T17:32:38.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7feed89ea640 1 -- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feecc0ca300 msgr2=0x7feecc0bd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7feed89ea640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feecc0ca300 0x7feecc0bd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7feed89ea640 1 -- 172.21.15.170:0/3136618199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feecc035f60 msgr2=0x7feecc0c6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7feed89ea640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feecc035f60 0x7feecc0c6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7feed89ea640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7feed89ea640 1 monclient: found mon.noname-a 2023-10-30T17:32:38.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7feed89ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7feed89ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:38.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7feed89ea640 1 -- 172.21.15.170:0/3136618199 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed4071040 con 0x7feecc039510 2023-10-30T17:32:38.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7f9517fff640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f952013eb60 0x7f95201431e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f950800c440 tx=0x7f950800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7f9516ffd640 10 client.?.objecter ms_handle_connect 0x7f952013eb60 2023-10-30T17:32:38.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.259+0000 7f9516ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:38.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.261+0000 7feed89ea640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feecc039510 0x7feecc0cad30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feed000e870 tx=0x7feed000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.261+0000 7feec9ffb640 10 client.?.objecter ms_handle_connect 0x7feecc039510 2023-10-30T17:32:38.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.261+0000 7feec9ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:38.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.267+0000 7f60417fa640 1 -- 172.21.15.170:0/1045790948 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f603800dd90 con 0x7f60440ae020 2023-10-30T17:32:38.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.267+0000 7f60417fa640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f603006c850 0x7f603006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.268+0000 7f6043fff640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f603006c850 0x7f603006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.268+0000 7f6043fff640 10 monclient: get_auth_request con 0x7f603006c850 auth_method 0 2023-10-30T17:32:38.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.268+0000 7f6043fff640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f603006c850 0x7f603006ed10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f604406d6d0 tx=0x7f603c007d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.268+0000 7f60417fa640 10 client.4297.objecter ms_handle_connect 0x7f603006c850 2023-10-30T17:32:38.492 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 1 -- 172.21.15.170:0/3136618199 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed000ee60 con 0x7feecc039510 2023-10-30T17:32:38.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:38.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 10 monclient: dump: 2023-10-30T17:32:38.493 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.493 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.494 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.494 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.494 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.494 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.495 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.495 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.495 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.495 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.495 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.495 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 1 -- 172.21.15.170:0/3136618199 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed000c940 con 0x7feecc039510 2023-10-30T17:32:38.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feeda1ed640 5 monclient: authenticate success, global_id 4311 2023-10-30T17:32:38.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feeda1ed640 10 monclient: _renew_subs 2023-10-30T17:32:38.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feeda1ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:38.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feeda1ed640 1 -- 172.21.15.170:0/3136618199 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feecc0bde10 con 0x7feecc039510 2023-10-30T17:32:38.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 1 -- 172.21.15.170:0/3136618199 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed000ccd0 con 0x7feecc039510 2023-10-30T17:32:38.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:38.498 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feec9ffb640 10 monclient: dump: 2023-10-30T17:32:38.498 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.498 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.498 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.498 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.499 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.499 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.499 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.499 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.499 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.499 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.270+0000 7feecb7fe640 4 set_mon_vals no callback set 2023-10-30T17:32:38.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feeda1ed640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feeda1ed640 10 monclient: _renew_subs 2023-10-30T17:32:38.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feeda1ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:38.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feeda1ed640 1 -- 172.21.15.170:0/3136618199 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feecc15cb20 con 0x7feecc039510 2023-10-30T17:32:38.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feeda1ed640 1 librados: init done 2023-10-30T17:32:38.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feebf7fe640 10 client.4311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:38.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feebf7fe640 15 client.4311.objecter _session_command_op_assign -1 1 2023-10-30T17:32:38.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feebf7fe640 20 client.4311.objecter _get_session osd=-1 returning homeless 2023-10-30T17:32:38.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feebf7fe640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feebf7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:38.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.271+0000 7feebf7fe640 1 -- 172.21.15.170:0/3136618199 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feeac0028a0 con 0x7feecc039510 2023-10-30T17:32:38.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f2c70cd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c6c0b0330 0x7f2c6c12c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 1 -- 172.21.15.170:0/1993976595 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95080035f0 con 0x7f952013eb60 2023-10-30T17:32:38.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:38.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 10 monclient: dump: 2023-10-30T17:32:38.504 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.504 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.505 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.505 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.505 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.505 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.505 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.506 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.506 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.506 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 1 -- 172.21.15.170:0/1993976595 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9508003770 con 0x7f952013eb60 2023-10-30T17:32:38.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 1 -- 172.21.15.170:0/1993976595 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9508003b20 con 0x7f952013eb60 2023-10-30T17:32:38.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9524cc9640 5 monclient: authenticate success, global_id 4308 2023-10-30T17:32:38.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9524cc9640 10 monclient: _renew_subs 2023-10-30T17:32:38.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9524cc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:38.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95201d6ac0 con 0x7f952013eb60 2023-10-30T17:32:38.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f951d59a640 4 set_mon_vals no callback set 2023-10-30T17:32:38.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:38.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9516ffd640 10 monclient: dump: 2023-10-30T17:32:38.509 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.509 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.509 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.509 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.510 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.510 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.510 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.510 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.510 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.511 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9524cc9640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9524cc9640 10 monclient: _renew_subs 2023-10-30T17:32:38.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9524cc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:38.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952013fa10 con 0x7f952013eb60 2023-10-30T17:32:38.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f9524cc9640 1 librados: init done 2023-10-30T17:32:38.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f2c70cd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c6c0b0330 0x7f2c6c12c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:39880/0 (socket says 172.21.15.170:39880) 2023-10-30T17:32:38.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f2c70cd0640 1 -- 172.21.15.170:0/1311993025 learned_addr learned my addr 172.21.15.170:0/1311993025 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f2c70cd0640 10 monclient(hunting): get_auth_request con 0x7f2c6c0b0330 auth_method 0 2023-10-30T17:32:38.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f2c70cd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f2c70cd0640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.272+0000 7f2c70cd0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.273+0000 7f9524cc9640 10 client.4308.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:38.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.273+0000 7f9524cc9640 15 client.4308.objecter _session_command_op_assign -1 1 2023-10-30T17:32:38.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.273+0000 7f9524cc9640 20 client.4308.objecter _get_session osd=-1 returning homeless 2023-10-30T17:32:38.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.273+0000 7f9524cc9640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.273+0000 7f9524cc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:38.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.273+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f95200a90c0 con 0x7f952013eb60 2023-10-30T17:32:38.515 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.273+0000 7f60417fa640 1 -- 172.21.15.170:0/1045790948 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f60380591b0 con 0x7f60440ae020 2023-10-30T17:32:38.515 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.273+0000 7f60417fa640 10 client.4297.objecter ms_dispatch 0x7f6044000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:38.515 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.273+0000 7f60417fa640 3 client.4297.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:38.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.273+0000 7f60417fa640 3 client.4297.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:38.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 10 client.4297.objecter checking command 1 2023-10-30T17:32:38.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] conn(0x7f6030072c30 0x7f6030075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f9495320640 1 --2- 172.21.15.170:0/2402540863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f949013d040 0x7f94900afae0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9488009d90 tx=0x7f948802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d94d5bd7693a5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 20 client.4297.objecter _get_session s=0x7f6030072560 osd=0 3 2023-10-30T17:32:38.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 20 client.4297.objecter put_session s=0x7f6030072560 osd=0 3 2023-10-30T17:32:38.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 15 client.4297.objecter _session_command_op_remove -1 1 2023-10-30T17:32:38.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 20 client.4297.objecter _get_session s=0x7f6030072560 osd=0 3 2023-10-30T17:32:38.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 20 client.4297.objecter get_session s=0x7f6030072560 osd=0 3 2023-10-30T17:32:38.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 15 client.4297.objecter _session_command_op_assign 0 1 2023-10-30T17:32:38.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 20 client.4297.objecter put_session s=0x7f6030072560 osd=0 4 2023-10-30T17:32:38.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 10 client.4297.objecter _send_command 1 2023-10-30T17:32:38.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 1 -- 172.21.15.170:0/1045790948 --> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6030075700 con 0x7f6030072c30 2023-10-30T17:32:38.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f60417fa640 20 client.4297.objecter dump_active .. 0 homeless 2023-10-30T17:32:38.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.274+0000 7f6049786640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] conn(0x7f6030072c30 0x7f6030075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.275+0000 7f6049786640 10 monclient: get_auth_request con 0x7f6030072c30 auth_method 0 2023-10-30T17:32:38.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.278+0000 7f60417fa640 10 client.4297.objecter ms_handle_connect 0x7f6030072c30 2023-10-30T17:32:38.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.278+0000 7f6049786640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] conn(0x7f6030072c30 0x7f6030075050 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.289+0000 7f9de2ffd640 1 -- 172.21.15.170:0/929183729 <== osd.4 v2:172.21.15.177:6802/2107882913 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f9dd40756b0 con 0x7f9dd4072be0 2023-10-30T17:32:38.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.289+0000 7f9de2ffd640 10 client.4268.objecter ms_dispatch 0x7f9df4000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:32:38.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.289+0000 7f9de2ffd640 10 client.4268.objecter _finish_command 1 = system:0 2023-10-30T17:32:38.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.289+0000 7f9de2ffd640 20 client.4268.objecter put_session s=0x7f9dd4072510 osd=4 4 2023-10-30T17:32:38.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.289+0000 7f9de2ffd640 15 client.4268.objecter _session_command_op_remove 4 1 2023-10-30T17:32:38.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 10 client.4268.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T17:32:38.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 15 client.4268.objecter _session_command_op_assign -1 2 2023-10-30T17:32:38.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 20 client.4268.objecter _get_session s=0x7f9dd4072510 osd=4 3 2023-10-30T17:32:38.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 20 client.4268.objecter put_session s=0x7f9dd4072510 osd=4 3 2023-10-30T17:32:38.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 20 client.4268.objecter _get_session s=0x7f9dd4072510 osd=4 3 2023-10-30T17:32:38.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 15 client.4268.objecter _session_command_op_remove -1 2 2023-10-30T17:32:38.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 20 client.4268.objecter get_session s=0x7f9dd4072510 osd=4 3 2023-10-30T17:32:38.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 15 client.4268.objecter _session_command_op_assign 4 2 2023-10-30T17:32:38.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 20 client.4268.objecter put_session s=0x7f9dd4072510 osd=4 4 2023-10-30T17:32:38.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 10 client.4268.objecter _send_command 2 2023-10-30T17:32:38.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.304+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 --> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9dec00a130 con 0x7f9dd4072be0 2023-10-30T17:32:38.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.309+0000 7f2c70cd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.309+0000 7f2c70cd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.309+0000 7f2c70cd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.309+0000 7f9516ffd640 1 -- 172.21.15.170:0/1993976595 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9508003c80 con 0x7f952013eb60 2023-10-30T17:32:38.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.310+0000 7f9516ffd640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f950006c800 0x7f950006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.310+0000 7f9de2ffd640 1 -- 172.21.15.170:0/929183729 <== osd.4 v2:172.21.15.177:6802/2107882913 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9dec00a130 con 0x7f9dd4072be0 2023-10-30T17:32:38.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.310+0000 7f9de2ffd640 10 client.4268.objecter ms_dispatch 0x7f9df4000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:32:38.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.310+0000 7f9de2ffd640 10 client.4268.objecter _finish_command 2 = system:0 2023-10-30T17:32:38.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.310+0000 7f9de2ffd640 20 client.4268.objecter put_session s=0x7f9dd4072510 osd=4 4 2023-10-30T17:32:38.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.310+0000 7f9de2ffd640 15 client.4268.objecter _session_command_op_remove 4 2 2023-10-30T17:32:38.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.310+0000 7f951e59c640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f950006c800 0x7f950006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.310+0000 7f951e59c640 10 monclient: get_auth_request con 0x7f950006c800 auth_method 0 2023-10-30T17:32:38.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.310+0000 7f9df8e4b640 10 librados: watch_flush enter 2023-10-30T17:32:38.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.310+0000 7f9df8e4b640 10 librados: watch_flush exit 2023-10-30T17:32:38.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 10 client.4268.objecter close_session for osd.4 2023-10-30T17:32:38.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] conn(0x7f9dd4072be0 msgr2=0x7f9dd4075000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9516ffd640 10 client.4308.objecter ms_handle_connect 0x7f950006c800 2023-10-30T17:32:38.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f951e59c640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f950006c800 0x7f950006ecc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9510036000 tx=0x7f9510030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] conn(0x7f9dd4072be0 0x7f9dd4075000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 20 client.4268.objecter put_session s=0x7f9dd4072510 osd=4 1 2023-10-30T17:32:38.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 20 client.4268.objecter shutdown clearing up homeless session... 2023-10-30T17:32:38.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 10 client.4268.objecter successfully canceled tick 2023-10-30T17:32:38.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9dd406c800 msgr2=0x7f9dd406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f9dd406c800 0x7f9dd406ecc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9de4006fd0 tx=0x7f9de4008040 comp rx=0 tx=0).stop 2023-10-30T17:32:38.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 10 monclient: shutdown 2023-10-30T17:32:38.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 20 monclient: shutdown canceling and discarding version request 1 2023-10-30T17:32:38.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dec039510 msgr2=0x7f9dec02fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df8e4b640 1 --2- 172.21.15.170:0/929183729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dec039510 0x7f9dec02fc90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9df4069460 tx=0x7f9df4061df0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.311+0000 7f9df2d9d640 1 -- 172.21.15.170:0/929183729 reap_dead start 2023-10-30T17:32:38.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.312+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 shutdown_connections 2023-10-30T17:32:38.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.312+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 >> 172.21.15.170:0/929183729 conn(0x7f9dec0070a0 msgr2=0x7f9dec03b040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.312+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 shutdown_connections 2023-10-30T17:32:38.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.312+0000 7f9df8e4b640 1 -- 172.21.15.170:0/929183729 wait complete. 2023-10-30T17:32:38.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.312+0000 7f9df8e4b640 1 librados: shutdown 2023-10-30T17:32:38.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.314+0000 7f2c6bfff640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c6c0a7220 0x7f2c6c071900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.315+0000 7f2c6bfff640 10 monclient(hunting): get_auth_request con 0x7f2c6c0a7220 auth_method 0 2023-10-30T17:32:38.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.315+0000 7f2c6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.315+0000 7f2c6bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.315+0000 7f2c6bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.316+0000 7f2c6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.316+0000 7f2c6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.316+0000 7f2c6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 1 -- 172.21.15.170:0/1993976595 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9508057fd0 con 0x7f952013eb60 2023-10-30T17:32:38.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 10 client.4308.objecter ms_dispatch 0x7f9520000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:38.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 3 client.4308.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:38.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 3 client.4308.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:38.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 10 client.4308.objecter checking command 1 2023-10-30T17:32:38.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] conn(0x7f9500072be0 0x7f9500075000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 20 client.4308.objecter _get_session s=0x7f9500072510 osd=2 3 2023-10-30T17:32:38.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 20 client.4308.objecter put_session s=0x7f9500072510 osd=2 3 2023-10-30T17:32:38.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 15 client.4308.objecter _session_command_op_remove -1 1 2023-10-30T17:32:38.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 20 client.4308.objecter _get_session s=0x7f9500072510 osd=2 3 2023-10-30T17:32:38.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 20 client.4308.objecter get_session s=0x7f9500072510 osd=2 3 2023-10-30T17:32:38.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 15 client.4308.objecter _session_command_op_assign 2 1 2023-10-30T17:32:38.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 20 client.4308.objecter put_session s=0x7f9500072510 osd=2 4 2023-10-30T17:32:38.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 10 client.4308.objecter _send_command 1 2023-10-30T17:32:38.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 1 -- 172.21.15.170:0/1993976595 --> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95000756b0 con 0x7f9500072be0 2023-10-30T17:32:38.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 20 client.4308.objecter dump_active .. 0 homeless 2023-10-30T17:32:38.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 1 -- 172.21.15.170:0/1993976595 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9508053c10 con 0x7f952013eb60 2023-10-30T17:32:38.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.318+0000 7f9516ffd640 10 monclient: handle_get_version_reply finishing 1 version 23 2023-10-30T17:32:38.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.319+0000 7f951ed9d640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] conn(0x7f9500072be0 0x7f9500075000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.319+0000 7f951ed9d640 10 monclient: get_auth_request con 0x7f9500072be0 auth_method 0 2023-10-30T17:32:38.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.321+0000 7f2c6bfff640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2023-10-30T17:32:38.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.321+0000 7f2c6bfff640 1 -- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6c0a8c00 msgr2=0x7f2c6c12df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.321+0000 7f2c6bfff640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6c0a8c00 0x7f2c6c12df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.321+0000 7f2c6bfff640 1 -- 172.21.15.170:0/1311993025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c6c0b0330 msgr2=0x7f2c6c12c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.321+0000 7f2c6bfff640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c6c0b0330 0x7f2c6c12c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.321+0000 7f2c6bfff640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.321+0000 7f2c6bfff640 1 monclient: found mon.noname-c 2023-10-30T17:32:38.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.321+0000 7f2c6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.321+0000 7f2c6bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.321+0000 7f2c6bfff640 1 -- 172.21.15.170:0/1311993025 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c64009a70 con 0x7f2c6c0a7220 2023-10-30T17:32:38.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.323+0000 7f2c6bfff640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c6c0a7220 0x7f2c6c071900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00c440 tx=0x7f2c5c00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.323+0000 7f2c68ff9640 10 client.?.objecter ms_handle_connect 0x7f2c6c0a7220 2023-10-30T17:32:38.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.323+0000 7f2c68ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:38.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.324+0000 7f9516ffd640 10 client.4308.objecter ms_handle_connect 0x7f9500072be0 2023-10-30T17:32:38.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.324+0000 7f951ed9d640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] conn(0x7f9500072be0 0x7f9500075000 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.545 INFO:teuthology.orchestra.run.smithi170.stdout:60129542151 2023-10-30T17:32:38.545 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T17:32:38.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 1 -- 172.21.15.170:0/1311993025 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c005ce0 con 0x7f2c6c0a7220 2023-10-30T17:32:38.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:38.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 10 monclient: dump: 2023-10-30T17:32:38.548 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.548 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.548 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.548 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.548 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.548 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.549 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.549 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.549 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.549 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c72733640 5 monclient: authenticate success, global_id 4280 2023-10-30T17:32:38.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c72733640 10 monclient: _renew_subs 2023-10-30T17:32:38.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c72733640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c6c12e480 con 0x7f2c6c0a7220 2023-10-30T17:32:38.551 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c72733640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.551 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c72733640 10 monclient: _renew_subs 2023-10-30T17:32:38.551 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c72733640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.551 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6c12b4c0 con 0x7f2c6c0a7220 2023-10-30T17:32:38.551 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c72733640 1 librados: init done 2023-10-30T17:32:38.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 1 -- 172.21.15.170:0/1311993025 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c005e40 con 0x7f2c6c0a7220 2023-10-30T17:32:38.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 1 -- 172.21.15.170:0/1311993025 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0072d0 con 0x7f2c6c0a7220 2023-10-30T17:32:38.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:38.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c68ff9640 10 monclient: dump: 2023-10-30T17:32:38.553 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.553 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.553 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.553 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.554 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.554 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.554 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.554 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.554 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.555 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.327+0000 7f2c6a7fc640 4 set_mon_vals no callback set 2023-10-30T17:32:38.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.328+0000 7f2c72733640 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:38.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.328+0000 7f2c72733640 15 client.4280.objecter _session_command_op_assign -1 1 2023-10-30T17:32:38.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.328+0000 7f2c72733640 20 client.4280.objecter _get_session osd=-1 returning homeless 2023-10-30T17:32:38.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.328+0000 7f2c72733640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.328+0000 7f2c72733640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.328+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c4c0028a0 con 0x7f2c6c0a7220 2023-10-30T17:32:38.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.332+0000 7f60417fa640 1 -- 172.21.15.170:0/1045790948 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6038007420 con 0x7f60440ae020 2023-10-30T17:32:38.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.332+0000 7f60417fa640 10 monclient: handle_get_version_reply finishing 1 version 23 2023-10-30T17:32:38.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.342+0000 7f60417fa640 1 -- 172.21.15.170:0/1045790948 <== osd.0 v2:172.21.15.170:6808/500152820 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f6030075700 con 0x7f6030072c30 2023-10-30T17:32:38.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.342+0000 7f60417fa640 10 client.4297.objecter ms_dispatch 0x7f6044000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:32:38.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.342+0000 7f60417fa640 10 client.4297.objecter _finish_command 1 = system:0 2023-10-30T17:32:38.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.342+0000 7f60417fa640 20 client.4297.objecter put_session s=0x7f6030072560 osd=0 4 2023-10-30T17:32:38.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.342+0000 7f60417fa640 15 client.4297.objecter _session_command_op_remove 0 1 2023-10-30T17:32:38.559 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 1 -- 172.21.15.170:0/2402540863 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948802cd00 con 0x7f949013d040 2023-10-30T17:32:38.559 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.559 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:38.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 10 monclient: dump: 2023-10-30T17:32:38.560 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.560 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.560 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.561 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.561 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.561 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.561 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.561 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.562 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.562 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.562 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.562 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 1 -- 172.21.15.170:0/2402540863 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948802ce60 con 0x7f949013d040 2023-10-30T17:32:38.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f9496d83640 5 monclient: authenticate success, global_id 4305 2023-10-30T17:32:38.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f9496d83640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:38.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f9496d83640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:38.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f9496d83640 4 set_mon_vals no callback set 2023-10-30T17:32:38.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 1 -- 172.21.15.170:0/2402540863 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488034660 con 0x7f949013d040 2023-10-30T17:32:38.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:38.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.344+0000 7f948f7fe640 10 monclient: dump: 2023-10-30T17:32:38.565 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.565 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.565 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.565 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.565 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.566 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.566 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.566 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.566 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.566 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.345+0000 7f9496d83640 10 monclient: shutdown 2023-10-30T17:32:38.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.345+0000 7f9496d83640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.345+0000 7f9496d83640 1 -- 172.21.15.170:0/2402540863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f949013d040 msgr2=0x7f94900afae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.345+0000 7f9496d83640 1 --2- 172.21.15.170:0/2402540863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f949013d040 0x7f94900afae0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9488009d90 tx=0x7f948802bf50 comp rx=0 tx=0).stop 2023-10-30T17:32:38.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.345+0000 7f9496d83640 1 -- 172.21.15.170:0/2402540863 shutdown_connections 2023-10-30T17:32:38.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.345+0000 7f9496d83640 1 --2- 172.21.15.170:0/2402540863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f949013d040 0x7f94900afae0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.345+0000 7f9496d83640 1 -- 172.21.15.170:0/2402540863 >> 172.21.15.170:0/2402540863 conn(0x7f94900b5340 msgr2=0x7f94900b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.345+0000 7f9496d83640 1 -- 172.21.15.170:0/2402540863 shutdown_connections 2023-10-30T17:32:38.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.345+0000 7f9496d83640 1 -- 172.21.15.170:0/2402540863 wait complete. 2023-10-30T17:32:38.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f3df97fa640 1 -- 172.21.15.170:0/1997246290 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8005ce0 con 0x7f3dfc13eb60 2023-10-30T17:32:38.569 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f3df97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.569 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f3df97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:38.569 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f3df97fa640 10 monclient: dump: 2023-10-30T17:32:38.569 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.569 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.569 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.570 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.570 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.570 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.570 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.570 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.571 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.571 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.571 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f3df97fa640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.571 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f3df97fa640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.571 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f3df97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f3e02b97640 5 monclient: authenticate success, global_id 4309 2023-10-30T17:32:38.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f3df97fa640 1 -- 172.21.15.170:0/1997246290 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de8005e40 con 0x7f3dfc13eb60 2023-10-30T17:32:38.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f9496d83640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f9496d83640 10 monclient: monmap: 2023-10-30T17:32:38.573 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.573 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:38.573 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:38.573 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:38.573 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.574 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.574 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:38.574 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:38.574 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:38.574 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.575 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.346+0000 7f3df97fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.575 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3df97fa640 1 -- 172.21.15.170:0/1997246290 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8005ce0 con 0x7f3dfc13eb60 2023-10-30T17:32:38.575 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3df97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.575 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3df97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:38.575 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3df97fa640 10 monclient: dump: 2023-10-30T17:32:38.576 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.576 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.576 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.576 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.576 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.577 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.577 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.577 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.577 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.577 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3dfaffd640 4 set_mon_vals no callback set 2023-10-30T17:32:38.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3e02b97640 10 monclient: _renew_subs 2023-10-30T17:32:38.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3e02b97640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc1435c0 con 0x7f3dfc13eb60 2023-10-30T17:32:38.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3e02b97640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3e02b97640 10 monclient: _renew_subs 2023-10-30T17:32:38.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3e02b97640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dfc1d6fd0 con 0x7f3dfc13eb60 2023-10-30T17:32:38.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f3e02b97640 1 librados: init done 2023-10-30T17:32:38.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f9496d83640 1 librados: starting msgr at 2023-10-30T17:32:38.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f9496d83640 1 librados: starting objecter 2023-10-30T17:32:38.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f39fdffb640 1 -- 172.21.15.170:0/784572801 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3a000043d0 con 0x7f3a0813daf0 2023-10-30T17:32:38.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f9496d83640 1 Processor -- start 2023-10-30T17:32:38.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f9496d83640 1 -- start start 2023-10-30T17:32:38.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f9496d83640 1 librados: setting wanted keys 2023-10-30T17:32:38.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f9496d83640 1 librados: calling monclient init 2023-10-30T17:32:38.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f9496d83640 10 monclient: init 2023-10-30T17:32:38.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.347+0000 7f39fdffb640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f39ec06c850 0x7f39ec06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f3e02b97640 10 client.4309.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:38.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f3e02b97640 15 client.4309.objecter _session_command_op_assign -1 1 2023-10-30T17:32:38.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f3e02b97640 20 client.4309.objecter _get_session osd=-1 returning homeless 2023-10-30T17:32:38.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f3e02b97640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f3e02b97640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3dfc0a90c0 con 0x7f3dfc13eb60 2023-10-30T17:32:38.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f3a0d473640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f39ec06c850 0x7f39ec06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f9496d83640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f9496d83640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:32:38.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f9496d83640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9490070100 0x7f9490135e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.348+0000 7f3a0d473640 10 monclient: get_auth_request con 0x7f39ec06c850 auth_method 0 2023-10-30T17:32:38.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 10 monclient(hunting): picked mon.noname-a con 0x7f9490070100 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:38.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94900704e0 0x7f94901d81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 10 monclient(hunting): picked mon.noname-b con 0x7f94900704e0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:38.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f949013d040 0x7f94901d8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.587 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 10 monclient(hunting): picked mon.noname-c con 0x7f949013d040 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.587 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 1 -- 172.21.15.170:0/784572801 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3a00059250 con 0x7f3a0813daf0 2023-10-30T17:32:38.588 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 10 client.4306.objecter ms_dispatch 0x7f3a08000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:38.588 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 3 client.4306.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:38.588 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 3 client.4306.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:38.588 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f3a0d473640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f39ec06c850 0x7f39ec06ed10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3a04039000 tx=0x7f3a04008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.589 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.589 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9490137920 con 0x7f9490070100 2023-10-30T17:32:38.589 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.590 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9490136440 con 0x7f94900704e0 2023-10-30T17:32:38.590 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.590 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9490134680 con 0x7f949013d040 2023-10-30T17:32:38.590 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.591 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 10 client.4306.objecter checking command 1 2023-10-30T17:32:38.591 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] conn(0x7f39ec072c30 0x7f39ec075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.591 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 20 client.4306.objecter _get_session s=0x7f39ec072560 osd=7 3 2023-10-30T17:32:38.591 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 20 client.4306.objecter put_session s=0x7f39ec072560 osd=7 3 2023-10-30T17:32:38.591 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 15 client.4306.objecter _session_command_op_remove -1 1 2023-10-30T17:32:38.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 20 client.4306.objecter _get_session s=0x7f39ec072560 osd=7 3 2023-10-30T17:32:38.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 20 client.4306.objecter get_session s=0x7f39ec072560 osd=7 3 2023-10-30T17:32:38.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 15 client.4306.objecter _session_command_op_assign 7 1 2023-10-30T17:32:38.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 20 client.4306.objecter put_session s=0x7f39ec072560 osd=7 4 2023-10-30T17:32:38.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 10 client.4306.objecter _send_command 1 2023-10-30T17:32:38.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 1 -- 172.21.15.170:0/784572801 --> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39ec075700 con 0x7f39ec072c30 2023-10-30T17:32:38.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 20 client.4306.objecter dump_active .. 0 homeless 2023-10-30T17:32:38.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 10 client.4306.objecter ms_handle_connect 0x7f39ec06c850 2023-10-30T17:32:38.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 1 -- 172.21.15.170:0/784572801 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a00055590 con 0x7f3a0813daf0 2023-10-30T17:32:38.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f39fdffb640 10 monclient: handle_get_version_reply finishing 1 version 23 2023-10-30T17:32:38.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.349+0000 7f9496d83640 10 monclient(hunting): authenticate will time out at 1340.672724s 2023-10-30T17:32:38.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.350+0000 7f3a0dc74640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] conn(0x7f39ec072c30 0x7f39ec075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.350+0000 7f3a0dc74640 10 monclient: get_auth_request con 0x7f39ec072c30 auth_method 0 2023-10-30T17:32:38.595 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.350+0000 7f948ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94900704e0 0x7f94901d81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.595 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.351+0000 7f948ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94900704e0 0x7f94901d81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60550/0 (socket says 172.21.15.170:60550) 2023-10-30T17:32:38.595 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.351+0000 7f948ffff640 1 -- 172.21.15.170:0/682459489 learned_addr learned my addr 172.21.15.170:0/682459489 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.595 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.351+0000 7f948ffff640 10 monclient(hunting): get_auth_request con 0x7f94900704e0 auth_method 0 2023-10-30T17:32:38.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.351+0000 7f948ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.351+0000 7f948ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.351+0000 7f948ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.354+0000 7f39fdffb640 10 client.4306.objecter ms_handle_connect 0x7f39ec072c30 2023-10-30T17:32:38.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.354+0000 7f3a0dc74640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] conn(0x7f39ec072c30 0x7f39ec075050 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.356+0000 7f2c68ff9640 1 -- 172.21.15.170:0/1311993025 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2c5c0074f0 con 0x7f2c6c0a7220 2023-10-30T17:32:38.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.356+0000 7f2c68ff9640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2c5806c800 0x7f2c5806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.356+0000 7f2c6b7fe640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2c5806c800 0x7f2c5806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.356+0000 7f2c6b7fe640 10 monclient: get_auth_request con 0x7f2c5806c800 auth_method 0 2023-10-30T17:32:38.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.357+0000 7f2c68ff9640 10 client.4280.objecter ms_handle_connect 0x7f2c5806c800 2023-10-30T17:32:38.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.357+0000 7f2c6b7fe640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2c5806c800 0x7f2c5806ecc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c60006a90 tx=0x7f2c60006a20 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 10 client.4297.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T17:32:38.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 15 client.4297.objecter _session_command_op_assign -1 2 2023-10-30T17:32:38.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 20 client.4297.objecter _get_session s=0x7f6030072560 osd=0 3 2023-10-30T17:32:38.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 20 client.4297.objecter put_session s=0x7f6030072560 osd=0 3 2023-10-30T17:32:38.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 20 client.4297.objecter _get_session s=0x7f6030072560 osd=0 3 2023-10-30T17:32:38.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 15 client.4297.objecter _session_command_op_remove -1 2 2023-10-30T17:32:38.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 20 client.4297.objecter get_session s=0x7f6030072560 osd=0 3 2023-10-30T17:32:38.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 15 client.4297.objecter _session_command_op_assign 0 2 2023-10-30T17:32:38.601 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 20 client.4297.objecter put_session s=0x7f6030072560 osd=0 4 2023-10-30T17:32:38.601 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 10 client.4297.objecter _send_command 2 2023-10-30T17:32:38.601 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.358+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 --> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f604413b4b0 con 0x7f6030072c30 2023-10-30T17:32:38.601 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.360+0000 7f948ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.360+0000 7f948ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.360+0000 7f948ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.604 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.360+0000 7f2c68ff9640 1 -- 172.21.15.170:0/1311993025 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2c5c0583a0 con 0x7f2c6c0a7220 2023-10-30T17:32:38.604 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.360+0000 7f2c68ff9640 10 client.4280.objecter ms_dispatch 0x7f2c6c000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:38.604 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.360+0000 7f2c68ff9640 3 client.4280.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:38.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.360+0000 7f2c68ff9640 3 client.4280.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:38.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 10 client.4280.objecter checking command 1 2023-10-30T17:32:38.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] conn(0x7f2c58072be0 0x7f2c58075000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 20 client.4280.objecter _get_session s=0x7f2c58072510 osd=5 3 2023-10-30T17:32:38.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 20 client.4280.objecter put_session s=0x7f2c58072510 osd=5 3 2023-10-30T17:32:38.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 15 client.4280.objecter _session_command_op_remove -1 1 2023-10-30T17:32:38.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 20 client.4280.objecter _get_session s=0x7f2c58072510 osd=5 3 2023-10-30T17:32:38.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 20 client.4280.objecter get_session s=0x7f2c58072510 osd=5 3 2023-10-30T17:32:38.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 15 client.4280.objecter _session_command_op_assign 5 1 2023-10-30T17:32:38.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 20 client.4280.objecter put_session s=0x7f2c58072510 osd=5 4 2023-10-30T17:32:38.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 10 client.4280.objecter _send_command 1 2023-10-30T17:32:38.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 1 -- 172.21.15.170:0/1311993025 --> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c580756b0 con 0x7f2c58072be0 2023-10-30T17:32:38.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 20 client.4280.objecter dump_active .. 0 homeless 2023-10-30T17:32:38.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 1 -- 172.21.15.170:0/1311993025 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c5c0546e0 con 0x7f2c6c0a7220 2023-10-30T17:32:38.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c68ff9640 10 monclient: handle_get_version_reply finishing 1 version 23 2023-10-30T17:32:38.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f9495320640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f949013d040 0x7f94901d8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.361+0000 7f2c70cd0640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] conn(0x7f2c58072be0 0x7f2c58075000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.362+0000 7f9495320640 10 monclient(hunting): get_auth_request con 0x7f949013d040 auth_method 0 2023-10-30T17:32:38.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.362+0000 7f9495320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.362+0000 7f9495320640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.362+0000 7f9495320640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.362+0000 7f2c70cd0640 10 monclient: get_auth_request con 0x7f2c58072be0 auth_method 0 2023-10-30T17:32:38.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.362+0000 7f9495320640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.362+0000 7f9495320640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.362+0000 7f9495320640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.363+0000 7feec9ffb640 1 -- 172.21.15.170:0/3136618199 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7feed0013470 con 0x7feecc039510 2023-10-30T17:32:38.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.364+0000 7feec9ffb640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7feeb806c880 0x7feeb806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.364+0000 7feed91eb640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7feeb806c880 0x7feeb806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.364+0000 7feed91eb640 10 monclient: get_auth_request con 0x7feeb806c880 auth_method 0 2023-10-30T17:32:38.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.364+0000 7feec9ffb640 10 client.4311.objecter ms_handle_connect 0x7feeb806c880 2023-10-30T17:32:38.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.364+0000 7feed91eb640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7feeb806c880 0x7feeb806ed40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feec40018b0 tx=0x7feec4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.365+0000 7f9495320640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2023-10-30T17:32:38.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.365+0000 7f9495320640 1 -- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94900704e0 msgr2=0x7f94901d81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.365+0000 7f9495320640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94900704e0 0x7f94901d81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.365+0000 7f9495320640 1 -- 172.21.15.170:0/682459489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9490070100 msgr2=0x7f9490135e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.365+0000 7f9495320640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9490070100 0x7f9490135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.365+0000 7f9495320640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.365+0000 7f9495320640 1 monclient: found mon.noname-c 2023-10-30T17:32:38.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.365+0000 7f9495320640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.365+0000 7f9495320640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.365+0000 7f9495320640 1 -- 172.21.15.170:0/682459489 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9488009a70 con 0x7f949013d040 2023-10-30T17:32:38.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.367+0000 7f2c70cd0640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] conn(0x7f2c58072be0 0x7f2c58075000 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.367+0000 7f2c68ff9640 10 client.4280.objecter ms_handle_connect 0x7f2c58072be0 2023-10-30T17:32:38.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.368+0000 7f9495320640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f949013d040 0x7f94901d8700 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9488033000 tx=0x7f9488004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.368+0000 7f948d7fa640 10 client.?.objecter ms_handle_connect 0x7f949013d040 2023-10-30T17:32:38.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.368+0000 7f948d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:38.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 1 -- 172.21.15.170:0/3136618199 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feed000ce30 con 0x7feecc039510 2023-10-30T17:32:38.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 10 client.4311.objecter ms_dispatch 0x7feed4000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:38.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 3 client.4311.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:38.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 3 client.4311.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:38.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 10 client.4311.objecter checking command 1 2023-10-30T17:32:38.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] conn(0x7feeb8072c60 0x7feeb8075080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 20 client.4311.objecter _get_session s=0x7feeb8072590 osd=1 3 2023-10-30T17:32:38.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 20 client.4311.objecter put_session s=0x7feeb8072590 osd=1 3 2023-10-30T17:32:38.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 15 client.4311.objecter _session_command_op_remove -1 1 2023-10-30T17:32:38.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 20 client.4311.objecter _get_session s=0x7feeb8072590 osd=1 3 2023-10-30T17:32:38.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 20 client.4311.objecter get_session s=0x7feeb8072590 osd=1 3 2023-10-30T17:32:38.620 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 15 client.4311.objecter _session_command_op_assign 1 1 2023-10-30T17:32:38.620 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 20 client.4311.objecter put_session s=0x7feeb8072590 osd=1 4 2023-10-30T17:32:38.620 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 10 client.4311.objecter _send_command 1 2023-10-30T17:32:38.620 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 1 -- 172.21.15.170:0/3136618199 --> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feeb8075730 con 0x7feeb8072c60 2023-10-30T17:32:38.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 20 client.4311.objecter dump_active .. 0 homeless 2023-10-30T17:32:38.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 1 -- 172.21.15.170:0/3136618199 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7feed0004640 con 0x7feecc039510 2023-10-30T17:32:38.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.369+0000 7feec9ffb640 10 monclient: handle_get_version_reply finishing 1 version 23 2023-10-30T17:32:38.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.370+0000 7feed99ec640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] conn(0x7feeb8072c60 0x7feeb8075080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f60417fa640 1 -- 172.21.15.170:0/1045790948 <== osd.0 v2:172.21.15.170:6808/500152820 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6034031a90 con 0x7f6030072c30 2023-10-30T17:32:38.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f60417fa640 10 client.4297.objecter ms_dispatch 0x7f6044000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:32:38.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f60417fa640 10 client.4297.objecter _finish_command 2 = system:0 2023-10-30T17:32:38.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f60417fa640 20 client.4297.objecter put_session s=0x7f6030072560 osd=0 4 2023-10-30T17:32:38.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f60417fa640 15 client.4297.objecter _session_command_op_remove 0 2 2023-10-30T17:32:38.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7feed99ec640 10 monclient: get_auth_request con 0x7feeb8072c60 auth_method 0 2023-10-30T17:32:38.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.370+0000 7f948d7fa640 1 -- 172.21.15.170:0/682459489 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948803d600 con 0x7f949013d040 2023-10-30T17:32:38.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.370+0000 7f948d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f948d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:38.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f948d7fa640 10 monclient: dump: 2023-10-30T17:32:38.624 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.624 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.624 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.624 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.625 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.625 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.625 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.625 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.625 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.626 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f948d7fa640 10 monclient: _finish_auth 0 2023-10-30T17:32:38.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f948d7fa640 10 monclient: _check_auth_tickets 2023-10-30T17:32:38.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f948d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:38.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f9496d83640 5 monclient: authenticate success, global_id 4283 2023-10-30T17:32:38.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f9496d83640 10 monclient: _renew_subs 2023-10-30T17:32:38.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f9496d83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94901352c0 con 0x7f949013d040 2023-10-30T17:32:38.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f6049f87640 10 librados: watch_flush enter 2023-10-30T17:32:38.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f6049f87640 10 librados: watch_flush exit 2023-10-30T17:32:38.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f6049f87640 10 client.4297.objecter close_session for osd.0 2023-10-30T17:32:38.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 >> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] conn(0x7f6030072c30 msgr2=0x7f6030075050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f6049f87640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] conn(0x7f6030072c30 0x7f6030075050 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f6049f87640 20 client.4297.objecter put_session s=0x7f6030072560 osd=0 1 2023-10-30T17:32:38.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f6049f87640 20 client.4297.objecter shutdown clearing up homeless session... 2023-10-30T17:32:38.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f6049f87640 10 client.4297.objecter successfully canceled tick 2023-10-30T17:32:38.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f603006c850 msgr2=0x7f603006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f603006c850 0x7f603006ed10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f604406d6d0 tx=0x7f603c007d10 comp rx=0 tx=0).stop 2023-10-30T17:32:38.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 10 monclient: shutdown 2023-10-30T17:32:38.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f9496d83640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60440ae020 msgr2=0x7f6044135620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 1 --2- 172.21.15.170:0/1045790948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60440ae020 0x7f6044135620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f603800a7a0 tx=0x7f603800ac60 comp rx=0 tx=0).stop 2023-10-30T17:32:38.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049786640 1 -- 172.21.15.170:0/1045790948 reap_dead start 2023-10-30T17:32:38.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f9496d83640 10 monclient: _renew_subs 2023-10-30T17:32:38.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f9496d83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9490137f20 con 0x7f949013d040 2023-10-30T17:32:38.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.371+0000 7f9496d83640 1 librados: init done 2023-10-30T17:32:38.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 shutdown_connections 2023-10-30T17:32:38.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 >> 172.21.15.170:0/1045790948 conn(0x7f60440afbe0 msgr2=0x7f60440b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f9496d83640 10 client.4283.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:38.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 shutdown_connections 2023-10-30T17:32:38.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f9496d83640 15 client.4283.objecter _session_command_op_assign -1 1 2023-10-30T17:32:38.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f9496d83640 20 client.4283.objecter _get_session osd=-1 returning homeless 2023-10-30T17:32:38.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 1 -- 172.21.15.170:0/1045790948 wait complete. 2023-10-30T17:32:38.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f9496d83640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:38.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f6049f87640 1 librados: shutdown 2023-10-30T17:32:38.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f9496d83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.372+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9490137a90 con 0x7f949013d040 2023-10-30T17:32:38.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.374+0000 7f3df97fa640 1 -- 172.21.15.170:0/1997246290 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3de8007070 con 0x7f3dfc13eb60 2023-10-30T17:32:38.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.374+0000 7f3df97fa640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3de006c860 0x7f3de006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.374+0000 7f3e00933640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3de006c860 0x7f3de006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.375+0000 7f3e00933640 10 monclient: get_auth_request con 0x7f3de006c860 auth_method 0 2023-10-30T17:32:38.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.375+0000 7f3df97fa640 10 client.4309.objecter ms_handle_connect 0x7f3de006c860 2023-10-30T17:32:38.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.375+0000 7f3e00933640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3de006c860 0x7f3de006ed20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009710 tx=0x7f3df0006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 1 -- 172.21.15.170:0/1997246290 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3de8057c90 con 0x7f3dfc13eb60 2023-10-30T17:32:38.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 10 client.4309.objecter ms_dispatch 0x7f3dfc000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:38.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 3 client.4309.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:38.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 3 client.4309.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:38.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 10 client.4309.objecter checking command 1 2023-10-30T17:32:38.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x7f3de0072c40 0x7f3de0075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 20 client.4309.objecter _get_session s=0x7f3de0072570 osd=3 3 2023-10-30T17:32:38.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 20 client.4309.objecter put_session s=0x7f3de0072570 osd=3 3 2023-10-30T17:32:38.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7feed99ec640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] conn(0x7feeb8072c60 0x7feeb8075080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 15 client.4309.objecter _session_command_op_remove -1 1 2023-10-30T17:32:38.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 20 client.4309.objecter _get_session s=0x7f3de0072570 osd=3 3 2023-10-30T17:32:38.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 20 client.4309.objecter get_session s=0x7f3de0072570 osd=3 3 2023-10-30T17:32:38.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 15 client.4309.objecter _session_command_op_assign 3 1 2023-10-30T17:32:38.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 20 client.4309.objecter put_session s=0x7f3de0072570 osd=3 4 2023-10-30T17:32:38.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 10 client.4309.objecter _send_command 1 2023-10-30T17:32:38.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 1 -- 172.21.15.170:0/1997246290 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3de0075710 con 0x7f3de0072c40 2023-10-30T17:32:38.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7f3df97fa640 20 client.4309.objecter dump_active .. 0 homeless 2023-10-30T17:32:38.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.376+0000 7feec9ffb640 10 client.4311.objecter ms_handle_connect 0x7feeb8072c60 2023-10-30T17:32:38.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.377+0000 7f3df97fa640 1 -- 172.21.15.170:0/1997246290 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3de8001200 con 0x7f3dfc13eb60 2023-10-30T17:32:38.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.377+0000 7f3df97fa640 10 monclient: handle_get_version_reply finishing 1 version 23 2023-10-30T17:32:38.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.377+0000 7f3e01134640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x7f3de0072c40 0x7f3de0075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.378+0000 7f3e01134640 10 monclient: get_auth_request con 0x7f3de0072c40 auth_method 0 2023-10-30T17:32:38.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.380+0000 7f9516ffd640 1 -- 172.21.15.170:0/1993976595 <== osd.2 v2:172.21.15.170:6816/425342180 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f95000756b0 con 0x7f9500072be0 2023-10-30T17:32:38.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.380+0000 7f9516ffd640 10 client.4308.objecter ms_dispatch 0x7f9520000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:32:38.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.380+0000 7f9516ffd640 10 client.4308.objecter _finish_command 1 = system:0 2023-10-30T17:32:38.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.380+0000 7f9516ffd640 20 client.4308.objecter put_session s=0x7f9500072510 osd=2 4 2023-10-30T17:32:38.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.380+0000 7f9516ffd640 15 client.4308.objecter _session_command_op_remove 2 1 2023-10-30T17:32:38.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.383+0000 7f3df97fa640 10 client.4309.objecter ms_handle_connect 0x7f3de0072c40 2023-10-30T17:32:38.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.383+0000 7f3e01134640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x7f3de0072c40 0x7f3de0075060 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.385+0000 7f948d7fa640 1 -- 172.21.15.170:0/682459489 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9490137a90 con 0x7f949013d040 2023-10-30T17:32:38.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.385+0000 7f948d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:38.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.385+0000 7f948d7fa640 1 -- 172.21.15.170:0/682459489 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948803d960 con 0x7f949013d040 2023-10-30T17:32:38.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.385+0000 7f948d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:38.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.385+0000 7f948d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:38.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.385+0000 7f948d7fa640 10 monclient: dump: 2023-10-30T17:32:38.649 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:38.649 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.650 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.650 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:38.650 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:38.650 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.650 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:38.651 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:38.651 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:38.651 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.385+0000 7f948effd640 4 set_mon_vals no callback set 2023-10-30T17:32:38.652 INFO:teuthology.orchestra.run.smithi170.stdout:73014444035 2023-10-30T17:32:38.653 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T17:32:38.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.401+0000 7f948d7fa640 1 -- 172.21.15.170:0/682459489 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f948803b040 con 0x7f949013d040 2023-10-30T17:32:38.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.401+0000 7f948d7fa640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f947406c800 0x7f947406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.402+0000 7f9494b1f640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f947406c800 0x7f947406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.402+0000 7f9494b1f640 10 monclient: get_auth_request con 0x7f947406c800 auth_method 0 2023-10-30T17:32:38.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.402+0000 7f9494b1f640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f947406c800 0x7f947406ecc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f94840015f0 tx=0x7f9484007a40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.402+0000 7f948d7fa640 10 client.4283.objecter ms_handle_connect 0x7f947406c800 2023-10-30T17:32:38.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 1 -- 172.21.15.170:0/682459489 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f948807aae0 con 0x7f949013d040 2023-10-30T17:32:38.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 10 client.4283.objecter ms_dispatch 0x7f9490000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:38.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 3 client.4283.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:38.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 3 client.4283.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:38.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 10 client.4283.objecter checking command 1 2023-10-30T17:32:38.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] conn(0x7f9474072be0 0x7f9474075000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 20 client.4283.objecter _get_session s=0x7f9474072510 osd=6 3 2023-10-30T17:32:38.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 20 client.4283.objecter put_session s=0x7f9474072510 osd=6 3 2023-10-30T17:32:38.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 15 client.4283.objecter _session_command_op_remove -1 1 2023-10-30T17:32:38.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 20 client.4283.objecter _get_session s=0x7f9474072510 osd=6 3 2023-10-30T17:32:38.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 20 client.4283.objecter get_session s=0x7f9474072510 osd=6 3 2023-10-30T17:32:38.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 15 client.4283.objecter _session_command_op_assign 6 1 2023-10-30T17:32:38.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 20 client.4283.objecter put_session s=0x7f9474072510 osd=6 4 2023-10-30T17:32:38.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 10 client.4283.objecter _send_command 1 2023-10-30T17:32:38.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 1 -- 172.21.15.170:0/682459489 --> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9474079ac0 con 0x7f9474072be0 2023-10-30T17:32:38.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 20 client.4283.objecter dump_active .. 0 homeless 2023-10-30T17:32:38.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.404+0000 7f948d7fa640 1 -- 172.21.15.170:0/682459489 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9488076e20 con 0x7f949013d040 2023-10-30T17:32:38.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.405+0000 7f948d7fa640 10 monclient: handle_get_version_reply finishing 1 version 23 2023-10-30T17:32:38.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.405+0000 7f948ffff640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] conn(0x7f9474072be0 0x7f9474075000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.405+0000 7f948ffff640 10 monclient: get_auth_request con 0x7f9474072be0 auth_method 0 2023-10-30T17:32:38.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 10 client.4308.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T17:32:38.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 15 client.4308.objecter _session_command_op_assign -1 2 2023-10-30T17:32:38.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 20 client.4308.objecter _get_session s=0x7f9500072510 osd=2 3 2023-10-30T17:32:38.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 20 client.4308.objecter put_session s=0x7f9500072510 osd=2 3 2023-10-30T17:32:38.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 20 client.4308.objecter _get_session s=0x7f9500072510 osd=2 3 2023-10-30T17:32:38.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 15 client.4308.objecter _session_command_op_remove -1 2 2023-10-30T17:32:38.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 20 client.4308.objecter get_session s=0x7f9500072510 osd=2 3 2023-10-30T17:32:38.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 15 client.4308.objecter _session_command_op_assign 2 2 2023-10-30T17:32:38.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 20 client.4308.objecter put_session s=0x7f9500072510 osd=2 4 2023-10-30T17:32:38.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 10 client.4308.objecter _send_command 2 2023-10-30T17:32:38.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.406+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 --> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f95200a9900 con 0x7f9500072be0 2023-10-30T17:32:38.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.407+0000 7f39fdffb640 1 -- 172.21.15.170:0/784572801 <== osd.7 v2:172.21.15.177:6818/3145923709 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f39ec075700 con 0x7f39ec072c30 2023-10-30T17:32:38.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.407+0000 7f39fdffb640 10 client.4306.objecter ms_dispatch 0x7f3a08000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:32:38.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.407+0000 7f39fdffb640 10 client.4306.objecter _finish_command 1 = system:0 2023-10-30T17:32:38.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.407+0000 7f39fdffb640 20 client.4306.objecter put_session s=0x7f39ec072560 osd=7 4 2023-10-30T17:32:38.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.407+0000 7f39fdffb640 15 client.4306.objecter _session_command_op_remove 7 1 2023-10-30T17:32:38.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.409+0000 7f948d7fa640 10 client.4283.objecter ms_handle_connect 0x7f9474072be0 2023-10-30T17:32:38.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.409+0000 7f948ffff640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] conn(0x7f9474072be0 0x7f9474075000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:38.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.413+0000 7f2c68ff9640 1 -- 172.21.15.170:0/1311993025 <== osd.5 v2:172.21.15.177:6808/2243649533 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f2c580756b0 con 0x7f2c58072be0 2023-10-30T17:32:38.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.413+0000 7f2c68ff9640 10 client.4280.objecter ms_dispatch 0x7f2c6c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:32:38.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.413+0000 7f2c68ff9640 10 client.4280.objecter _finish_command 1 = system:0 2023-10-30T17:32:38.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.413+0000 7f2c68ff9640 20 client.4280.objecter put_session s=0x7f2c58072510 osd=5 4 2023-10-30T17:32:38.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.413+0000 7f2c68ff9640 15 client.4280.objecter _session_command_op_remove 5 1 2023-10-30T17:32:38.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9516ffd640 1 -- 172.21.15.170:0/1993976595 <== osd.2 v2:172.21.15.170:6816/425342180 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f951800f040 con 0x7f9500072be0 2023-10-30T17:32:38.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9516ffd640 10 client.4308.objecter ms_dispatch 0x7f9520000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:32:38.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9516ffd640 10 client.4308.objecter _finish_command 2 = system:0 2023-10-30T17:32:38.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9516ffd640 20 client.4308.objecter put_session s=0x7f9500072510 osd=2 4 2023-10-30T17:32:38.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9516ffd640 15 client.4308.objecter _session_command_op_remove 2 2 2023-10-30T17:32:38.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9524cc9640 10 librados: watch_flush enter 2023-10-30T17:32:38.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9524cc9640 10 librados: watch_flush exit 2023-10-30T17:32:38.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9524cc9640 10 client.4308.objecter close_session for osd.2 2023-10-30T17:32:38.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 >> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] conn(0x7f9500072be0 msgr2=0x7f9500075000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9524cc9640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] conn(0x7f9500072be0 0x7f9500075000 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9524cc9640 20 client.4308.objecter put_session s=0x7f9500072510 osd=2 1 2023-10-30T17:32:38.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9524cc9640 20 client.4308.objecter shutdown clearing up homeless session... 2023-10-30T17:32:38.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9524cc9640 10 client.4308.objecter successfully canceled tick 2023-10-30T17:32:38.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.415+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f950006c800 msgr2=0x7f950006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.416+0000 7f9524cc9640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f950006c800 0x7f950006ecc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9510036000 tx=0x7f9510030000 comp rx=0 tx=0).stop 2023-10-30T17:32:38.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.416+0000 7f9524cc9640 10 monclient: shutdown 2023-10-30T17:32:38.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.416+0000 7f9524cc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.416+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f952013eb60 msgr2=0x7f95201431e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.416+0000 7f9524cc9640 1 --2- 172.21.15.170:0/1993976595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f952013eb60 0x7f95201431e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f950800c440 tx=0x7f950800c900 comp rx=0 tx=0).stop 2023-10-30T17:32:38.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.416+0000 7f951ed9d640 1 -- 172.21.15.170:0/1993976595 reap_dead start 2023-10-30T17:32:38.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.416+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 shutdown_connections 2023-10-30T17:32:38.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.416+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 >> 172.21.15.170:0/1993976595 conn(0x7f9520071da0 msgr2=0x7f95200b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.416+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 shutdown_connections 2023-10-30T17:32:38.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.416+0000 7f9524cc9640 1 -- 172.21.15.170:0/1993976595 wait complete. 2023-10-30T17:32:38.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.417+0000 7f9524cc9640 1 librados: shutdown 2023-10-30T17:32:38.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7feec9ffb640 1 -- 172.21.15.170:0/3136618199 <== osd.1 v2:172.21.15.170:6824/527201623 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7feeb8075730 con 0x7feeb8072c60 2023-10-30T17:32:38.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7feec9ffb640 10 client.4311.objecter ms_dispatch 0x7feed4000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:32:38.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7feec9ffb640 10 client.4311.objecter _finish_command 1 = system:0 2023-10-30T17:32:38.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7feec9ffb640 20 client.4311.objecter put_session s=0x7feeb8072590 osd=1 4 2023-10-30T17:32:38.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7feec9ffb640 15 client.4311.objecter _session_command_op_remove 1 1 2023-10-30T17:32:38.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 10 client.4306.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T17:32:38.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 15 client.4306.objecter _session_command_op_assign -1 2 2023-10-30T17:32:38.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 20 client.4306.objecter _get_session s=0x7f39ec072560 osd=7 3 2023-10-30T17:32:38.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 20 client.4306.objecter put_session s=0x7f39ec072560 osd=7 3 2023-10-30T17:32:38.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 20 client.4306.objecter _get_session s=0x7f39ec072560 osd=7 3 2023-10-30T17:32:38.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 15 client.4306.objecter _session_command_op_remove -1 2 2023-10-30T17:32:38.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 20 client.4306.objecter get_session s=0x7f39ec072560 osd=7 3 2023-10-30T17:32:38.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 15 client.4306.objecter _session_command_op_assign 7 2 2023-10-30T17:32:38.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 20 client.4306.objecter put_session s=0x7f39ec072560 osd=7 4 2023-10-30T17:32:38.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 10 client.4306.objecter _send_command 2 2023-10-30T17:32:38.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.422+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 --> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a080aba50 con 0x7f39ec072c30 2023-10-30T17:32:38.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f39fdffb640 1 -- 172.21.15.170:0/784572801 <== osd.7 v2:172.21.15.177:6818/3145923709 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f39f80014e0 con 0x7f39ec072c30 2023-10-30T17:32:38.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f39fdffb640 10 client.4306.objecter ms_dispatch 0x7f3a08000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:32:38.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f39fdffb640 10 client.4306.objecter _finish_command 2 = system:0 2023-10-30T17:32:38.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f39fdffb640 20 client.4306.objecter put_session s=0x7f39ec072560 osd=7 4 2023-10-30T17:32:38.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f39fdffb640 15 client.4306.objecter _session_command_op_remove 7 2 2023-10-30T17:32:38.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f3a0e475640 10 librados: watch_flush enter 2023-10-30T17:32:38.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f3a0e475640 10 librados: watch_flush exit 2023-10-30T17:32:38.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f3a0e475640 10 client.4306.objecter close_session for osd.7 2023-10-30T17:32:38.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] conn(0x7f39ec072c30 msgr2=0x7f39ec075050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f3a0e475640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] conn(0x7f39ec072c30 0x7f39ec075050 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f3a0e475640 20 client.4306.objecter put_session s=0x7f39ec072560 osd=7 1 2023-10-30T17:32:38.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f3a0e475640 20 client.4306.objecter shutdown clearing up homeless session... 2023-10-30T17:32:38.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.429+0000 7f3a0e475640 10 client.4306.objecter successfully canceled tick 2023-10-30T17:32:38.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f39ec06c850 msgr2=0x7f39ec06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f39ec06c850 0x7f39ec06ed10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3a04039000 tx=0x7f3a04008040 comp rx=0 tx=0).stop 2023-10-30T17:32:38.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 10 monclient: shutdown 2023-10-30T17:32:38.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a0813daf0 msgr2=0x7f3a081393e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 1 --2- 172.21.15.170:0/784572801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a0813daf0 0x7f3a081393e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3a00007e60 tx=0x7f3a0000e4b0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0dc74640 1 -- 172.21.15.170:0/784572801 reap_dead start 2023-10-30T17:32:38.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 shutdown_connections 2023-10-30T17:32:38.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 >> 172.21.15.170:0/784572801 conn(0x7f3a080afbe0 msgr2=0x7f3a080b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 shutdown_connections 2023-10-30T17:32:38.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 1 -- 172.21.15.170:0/784572801 wait complete. 2023-10-30T17:32:38.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.430+0000 7f3a0e475640 1 librados: shutdown 2023-10-30T17:32:38.682 INFO:teuthology.orchestra.run.smithi170.stdout:64424509444 2023-10-30T17:32:38.682 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T17:32:38.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 10 client.4311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T17:32:38.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 15 client.4311.objecter _session_command_op_assign -1 2 2023-10-30T17:32:38.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 20 client.4311.objecter _get_session s=0x7feeb8072590 osd=1 3 2023-10-30T17:32:38.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 20 client.4311.objecter put_session s=0x7feeb8072590 osd=1 3 2023-10-30T17:32:38.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 20 client.4311.objecter _get_session s=0x7feeb8072590 osd=1 3 2023-10-30T17:32:38.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 15 client.4311.objecter _session_command_op_remove -1 2 2023-10-30T17:32:38.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 20 client.4311.objecter get_session s=0x7feeb8072590 osd=1 3 2023-10-30T17:32:38.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 15 client.4311.objecter _session_command_op_assign 1 2 2023-10-30T17:32:38.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 20 client.4311.objecter put_session s=0x7feeb8072590 osd=1 4 2023-10-30T17:32:38.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 10 client.4311.objecter _send_command 2 2023-10-30T17:32:38.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.440+0000 7feebf7fe640 1 -- 172.21.15.170:0/3136618199 --> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feeac0011c0 con 0x7feeb8072c60 2023-10-30T17:32:38.686 INFO:teuthology.orchestra.run.smithi170.stdout:60129542148 2023-10-30T17:32:38.686 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T17:32:38.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T17:32:38.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 15 client.4280.objecter _session_command_op_assign -1 2 2023-10-30T17:32:38.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 20 client.4280.objecter _get_session s=0x7f2c58072510 osd=5 3 2023-10-30T17:32:38.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 20 client.4280.objecter put_session s=0x7f2c58072510 osd=5 3 2023-10-30T17:32:38.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 20 client.4280.objecter _get_session s=0x7f2c58072510 osd=5 3 2023-10-30T17:32:38.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 15 client.4280.objecter _session_command_op_remove -1 2 2023-10-30T17:32:38.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 20 client.4280.objecter get_session s=0x7f2c58072510 osd=5 3 2023-10-30T17:32:38.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 15 client.4280.objecter _session_command_op_assign 5 2 2023-10-30T17:32:38.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 20 client.4280.objecter put_session s=0x7f2c58072510 osd=5 4 2023-10-30T17:32:38.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 10 client.4280.objecter _send_command 2 2023-10-30T17:32:38.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.441+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 --> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2c4c0011c0 con 0x7f2c58072be0 2023-10-30T17:32:38.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feec9ffb640 1 -- 172.21.15.170:0/3136618199 <== osd.1 v2:172.21.15.170:6824/527201623 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feeac0011c0 con 0x7feeb8072c60 2023-10-30T17:32:38.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feec9ffb640 10 client.4311.objecter ms_dispatch 0x7feed4000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:32:38.690 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feec9ffb640 10 client.4311.objecter _finish_command 2 = system:0 2023-10-30T17:32:38.690 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feec9ffb640 20 client.4311.objecter put_session s=0x7feeb8072590 osd=1 4 2023-10-30T17:32:38.690 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feec9ffb640 15 client.4311.objecter _session_command_op_remove 1 2 2023-10-30T17:32:38.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 10 librados: watch_flush enter 2023-10-30T17:32:38.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 10 librados: watch_flush exit 2023-10-30T17:32:38.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 10 client.4311.objecter close_session for osd.1 2023-10-30T17:32:38.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 -- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] conn(0x7feeb8072c60 msgr2=0x7feeb8075080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] conn(0x7feeb8072c60 0x7feeb8075080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 20 client.4311.objecter put_session s=0x7feeb8072590 osd=1 1 2023-10-30T17:32:38.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 20 client.4311.objecter shutdown clearing up homeless session... 2023-10-30T17:32:38.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 10 client.4311.objecter successfully canceled tick 2023-10-30T17:32:38.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 -- 172.21.15.170:0/3136618199 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7feeb806c880 msgr2=0x7feeb806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7feeb806c880 0x7feeb806ed40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feec40018b0 tx=0x7feec4009040 comp rx=0 tx=0).stop 2023-10-30T17:32:38.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 10 monclient: shutdown 2023-10-30T17:32:38.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 -- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feecc039510 msgr2=0x7feecc0cad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 --2- 172.21.15.170:0/3136618199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feecc039510 0x7feecc0cad30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feed000e870 tx=0x7feed000ed30 comp rx=0 tx=0).stop 2023-10-30T17:32:38.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feed99ec640 1 -- 172.21.15.170:0/3136618199 reap_dead start 2023-10-30T17:32:38.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 -- 172.21.15.170:0/3136618199 shutdown_connections 2023-10-30T17:32:38.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 -- 172.21.15.170:0/3136618199 >> 172.21.15.170:0/3136618199 conn(0x7feecc0070a0 msgr2=0x7feecc0c3630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 -- 172.21.15.170:0/3136618199 shutdown_connections 2023-10-30T17:32:38.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 -- 172.21.15.170:0/3136618199 wait complete. 2023-10-30T17:32:38.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.446+0000 7feebf7fe640 1 librados: shutdown 2023-10-30T17:32:38.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c68ff9640 1 -- 172.21.15.170:0/1311993025 <== osd.5 v2:172.21.15.177:6808/2243649533 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2c4c0011c0 con 0x7f2c58072be0 2023-10-30T17:32:38.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c68ff9640 10 client.4280.objecter ms_dispatch 0x7f2c6c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:32:38.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c68ff9640 10 client.4280.objecter _finish_command 2 = system:0 2023-10-30T17:32:38.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c68ff9640 20 client.4280.objecter put_session s=0x7f2c58072510 osd=5 4 2023-10-30T17:32:38.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c68ff9640 15 client.4280.objecter _session_command_op_remove 5 2 2023-10-30T17:32:38.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 10 librados: watch_flush enter 2023-10-30T17:32:38.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 10 librados: watch_flush exit 2023-10-30T17:32:38.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 10 client.4280.objecter close_session for osd.5 2023-10-30T17:32:38.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] conn(0x7f2c58072be0 msgr2=0x7f2c58075000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] conn(0x7f2c58072be0 0x7f2c58075000 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 20 client.4280.objecter put_session s=0x7f2c58072510 osd=5 1 2023-10-30T17:32:38.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 20 client.4280.objecter shutdown clearing up homeless session... 2023-10-30T17:32:38.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 10 client.4280.objecter successfully canceled tick 2023-10-30T17:32:38.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2c5806c800 msgr2=0x7f2c5806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2c5806c800 0x7f2c5806ecc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c60006a90 tx=0x7f2c60006a20 comp rx=0 tx=0).stop 2023-10-30T17:32:38.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 10 monclient: shutdown 2023-10-30T17:32:38.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c6c0a7220 msgr2=0x7f2c6c071900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 --2- 172.21.15.170:0/1311993025 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c6c0a7220 0x7f2c6c071900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00c440 tx=0x7f2c5c00c900 comp rx=0 tx=0).stop 2023-10-30T17:32:38.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c70cd0640 1 -- 172.21.15.170:0/1311993025 reap_dead start 2023-10-30T17:32:38.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 shutdown_connections 2023-10-30T17:32:38.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 >> 172.21.15.170:0/1311993025 conn(0x7f2c6c0b3560 msgr2=0x7f2c6c0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 shutdown_connections 2023-10-30T17:32:38.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 -- 172.21.15.170:0/1311993025 wait complete. 2023-10-30T17:32:38.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.448+0000 7f2c72733640 1 librados: shutdown 2023-10-30T17:32:38.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.450+0000 7f3df97fa640 1 -- 172.21.15.170:0/1997246290 <== osd.3 v2:172.21.15.170:6800/3657546070 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f3de0075710 con 0x7f3de0072c40 2023-10-30T17:32:38.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.450+0000 7f3df97fa640 10 client.4309.objecter ms_dispatch 0x7f3dfc000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:32:38.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.450+0000 7f3df97fa640 10 client.4309.objecter _finish_command 1 = system:0 2023-10-30T17:32:38.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.450+0000 7f3df97fa640 20 client.4309.objecter put_session s=0x7f3de0072570 osd=3 4 2023-10-30T17:32:38.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.450+0000 7f3df97fa640 15 client.4309.objecter _session_command_op_remove 3 1 2023-10-30T17:32:38.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.456+0000 7f948d7fa640 1 -- 172.21.15.170:0/682459489 <== osd.6 v2:172.21.15.177:6826/1041798931 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f9474079ac0 con 0x7f9474072be0 2023-10-30T17:32:38.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.457+0000 7f948d7fa640 10 client.4283.objecter ms_dispatch 0x7f9490000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:32:38.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.457+0000 7f948d7fa640 10 client.4283.objecter _finish_command 1 = system:0 2023-10-30T17:32:38.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.457+0000 7f948d7fa640 20 client.4283.objecter put_session s=0x7f9474072510 osd=6 4 2023-10-30T17:32:38.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.457+0000 7f948d7fa640 15 client.4283.objecter _session_command_op_remove 6 1 2023-10-30T17:32:38.705 INFO:teuthology.orchestra.run.smithi170.stdout:68719476742 2023-10-30T17:32:38.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T17:32:38.707 INFO:teuthology.orchestra.run.smithi170.stdout:55834574854 2023-10-30T17:32:38.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T17:32:38.708 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 10 client.4309.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T17:32:38.708 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 15 client.4309.objecter _session_command_op_assign -1 2 2023-10-30T17:32:38.708 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 20 client.4309.objecter _get_session s=0x7f3de0072570 osd=3 3 2023-10-30T17:32:38.708 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 20 client.4309.objecter put_session s=0x7f3de0072570 osd=3 3 2023-10-30T17:32:38.709 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 20 client.4309.objecter _get_session s=0x7f3de0072570 osd=3 3 2023-10-30T17:32:38.709 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 15 client.4309.objecter _session_command_op_remove -1 2 2023-10-30T17:32:38.709 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 20 client.4309.objecter get_session s=0x7f3de0072570 osd=3 3 2023-10-30T17:32:38.709 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 15 client.4309.objecter _session_command_op_assign 3 2 2023-10-30T17:32:38.709 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 20 client.4309.objecter put_session s=0x7f3de0072570 osd=3 4 2023-10-30T17:32:38.710 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 10 client.4309.objecter _send_command 2 2023-10-30T17:32:38.710 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.474+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3dfc0a9900 con 0x7f3de0072c40 2023-10-30T17:32:38.710 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 10 client.4283.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T17:32:38.711 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 15 client.4283.objecter _session_command_op_assign -1 2 2023-10-30T17:32:38.711 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 20 client.4283.objecter _get_session s=0x7f9474072510 osd=6 3 2023-10-30T17:32:38.711 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 20 client.4283.objecter put_session s=0x7f9474072510 osd=6 3 2023-10-30T17:32:38.711 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 20 client.4283.objecter _get_session s=0x7f9474072510 osd=6 3 2023-10-30T17:32:38.711 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 15 client.4283.objecter _session_command_op_remove -1 2 2023-10-30T17:32:38.711 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 20 client.4283.objecter get_session s=0x7f9474072510 osd=6 3 2023-10-30T17:32:38.712 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 15 client.4283.objecter _session_command_op_assign 6 2 2023-10-30T17:32:38.712 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 20 client.4283.objecter put_session s=0x7f9474072510 osd=6 4 2023-10-30T17:32:38.712 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 10 client.4283.objecter _send_command 2 2023-10-30T17:32:38.712 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.480+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 --> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f949013faf0 con 0x7f9474072be0 2023-10-30T17:32:38.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.481+0000 7f3df97fa640 1 -- 172.21.15.170:0/1997246290 <== osd.3 v2:172.21.15.170:6800/3657546070 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3df402ce90 con 0x7f3de0072c40 2023-10-30T17:32:38.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.481+0000 7f3df97fa640 10 client.4309.objecter ms_dispatch 0x7f3dfc000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:32:38.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.481+0000 7f3df97fa640 10 client.4309.objecter _finish_command 2 = system:0 2023-10-30T17:32:38.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.481+0000 7f3df97fa640 20 client.4309.objecter put_session s=0x7f3de0072570 osd=3 4 2023-10-30T17:32:38.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.481+0000 7f3df97fa640 15 client.4309.objecter _session_command_op_remove 3 2 2023-10-30T17:32:38.715 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 10 librados: watch_flush enter 2023-10-30T17:32:38.715 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 10 librados: watch_flush exit 2023-10-30T17:32:38.715 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 10 client.4309.objecter close_session for osd.3 2023-10-30T17:32:38.716 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x7f3de0072c40 msgr2=0x7f3de0075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.716 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x7f3de0072c40 0x7f3de0075060 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.716 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 20 client.4309.objecter put_session s=0x7f3de0072570 osd=3 1 2023-10-30T17:32:38.716 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 20 client.4309.objecter shutdown clearing up homeless session... 2023-10-30T17:32:38.716 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 10 client.4309.objecter successfully canceled tick 2023-10-30T17:32:38.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3de006c860 msgr2=0x7f3de006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3de006c860 0x7f3de006ed20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009710 tx=0x7f3df0006c90 comp rx=0 tx=0).stop 2023-10-30T17:32:38.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 10 monclient: shutdown 2023-10-30T17:32:38.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dfc13eb60 msgr2=0x7f3dfc0ab360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.718 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 --2- 172.21.15.170:0/1997246290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dfc13eb60 0x7f3dfc0ab360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3de800c4d0 tx=0x7f3de800c990 comp rx=0 tx=0).stop 2023-10-30T17:32:38.718 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e01134640 1 -- 172.21.15.170:0/1997246290 reap_dead start 2023-10-30T17:32:38.718 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 shutdown_connections 2023-10-30T17:32:38.718 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 >> 172.21.15.170:0/1997246290 conn(0x7f3dfc071da0 msgr2=0x7f3dfc0b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.718 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 shutdown_connections 2023-10-30T17:32:38.719 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 -- 172.21.15.170:0/1997246290 wait complete. 2023-10-30T17:32:38.719 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.482+0000 7f3e02b97640 1 librados: shutdown 2023-10-30T17:32:38.719 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f948d7fa640 1 -- 172.21.15.170:0/682459489 <== osd.6 v2:172.21.15.177:6826/1041798931 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f947c001560 con 0x7f9474072be0 2023-10-30T17:32:38.720 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f948d7fa640 10 client.4283.objecter ms_dispatch 0x7f9490000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:32:38.720 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f948d7fa640 10 client.4283.objecter _finish_command 2 = system:0 2023-10-30T17:32:38.720 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f948d7fa640 20 client.4283.objecter put_session s=0x7f9474072510 osd=6 4 2023-10-30T17:32:38.720 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f948d7fa640 15 client.4283.objecter _session_command_op_remove 6 2 2023-10-30T17:32:38.723 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 10 librados: watch_flush enter 2023-10-30T17:32:38.723 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 10 librados: watch_flush exit 2023-10-30T17:32:38.723 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 10 client.4283.objecter close_session for osd.6 2023-10-30T17:32:38.723 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] conn(0x7f9474072be0 msgr2=0x7f9474075000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.723 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] conn(0x7f9474072be0 0x7f9474075000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.724 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 20 client.4283.objecter put_session s=0x7f9474072510 osd=6 1 2023-10-30T17:32:38.724 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 20 client.4283.objecter shutdown clearing up homeless session... 2023-10-30T17:32:38.724 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 10 client.4283.objecter successfully canceled tick 2023-10-30T17:32:38.724 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f947406c800 msgr2=0x7f947406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.724 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f947406c800 0x7f947406ecc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f94840015f0 tx=0x7f9484007a40 comp rx=0 tx=0).stop 2023-10-30T17:32:38.725 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 10 monclient: shutdown 2023-10-30T17:32:38.725 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:38.725 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f949013d040 msgr2=0x7f94901d8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.725 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 --2- 172.21.15.170:0/682459489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f949013d040 0x7f94901d8700 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9488033000 tx=0x7f9488004a40 comp rx=0 tx=0).stop 2023-10-30T17:32:38.725 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9495320640 1 -- 172.21.15.170:0/682459489 reap_dead start 2023-10-30T17:32:38.726 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 shutdown_connections 2023-10-30T17:32:38.726 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 >> 172.21.15.170:0/682459489 conn(0x7f94900b5340 msgr2=0x7f94900b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:38.726 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 shutdown_connections 2023-10-30T17:32:38.726 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 -- 172.21.15.170:0/682459489 wait complete. 2023-10-30T17:32:38.726 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.486+0000 7f9496d83640 1 librados: shutdown 2023-10-30T17:32:38.727 INFO:teuthology.orchestra.run.smithi170.stdout:68719476740 2023-10-30T17:32:38.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T17:32:38.729 INFO:teuthology.orchestra.run.smithi170.stdout:60129542150 2023-10-30T17:32:38.729 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T17:32:38.918 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.916+0000 7ff9e4a21640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.918 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.916+0000 7ff9e4a21640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.916+0000 7ff9e4a21640 10 monclient: monmap: 2023-10-30T17:32:38.919 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.919 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.919 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.917731+0000 2023-10-30T17:32:38.919 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.917731+0000 2023-10-30T17:32:38.920 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.920 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.920 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.920 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.921 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.921 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.921 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.921 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.922 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.918+0000 7efd6f6d7640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.922 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.918+0000 7efd6f6d7640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.918+0000 7efd6f6d7640 10 monclient: monmap: 2023-10-30T17:32:38.923 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.923 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.923 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.919070+0000 2023-10-30T17:32:38.923 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.919070+0000 2023-10-30T17:32:38.924 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.924 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.924 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.924 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.925 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.925 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.925 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.925 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.917+0000 7f5749483640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.917+0000 7f5749483640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.917+0000 7f5749483640 10 monclient: monmap: 2023-10-30T17:32:38.926 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.927 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.927 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.918740+0000 2023-10-30T17:32:38.927 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.918740+0000 2023-10-30T17:32:38.927 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.928 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.928 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.928 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.928 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.928 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.928 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.929 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.917+0000 7face4cf2640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.917+0000 7face4cf2640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.930 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.917+0000 7face4cf2640 10 monclient: monmap: 2023-10-30T17:32:38.930 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.930 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.930 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.918046+0000 2023-10-30T17:32:38.930 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.918046+0000 2023-10-30T17:32:38.931 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.931 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.931 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.931 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.931 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.932 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.932 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.932 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.933 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.918+0000 7f5749483640 1 Processor -- start 2023-10-30T17:32:38.933 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.918+0000 7face4cf2640 1 Processor -- start 2023-10-30T17:32:38.933 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.918+0000 7face4cf2640 1 -- start start 2023-10-30T17:32:38.934 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.918+0000 7face4cf2640 10 monclient: init 2023-10-30T17:32:38.934 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.918+0000 7efd6f6d7640 1 Processor -- start 2023-10-30T17:32:38.934 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.918+0000 7f5749483640 1 -- start start 2023-10-30T17:32:38.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.918+0000 7f5749483640 10 monclient: init 2023-10-30T17:32:38.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7efd6f6d7640 1 -- start start 2023-10-30T17:32:38.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7efd6f6d7640 10 monclient: init 2023-10-30T17:32:38.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-10-30T17:32:38.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f57440aaed0 0x7f57440a9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 10 monclient(hunting): picked mon.noname-a con 0x7f57440aaed0 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:38.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 0x7f57440a9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 10 monclient(hunting): picked mon.noname-b con 0x7f57440a65a0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57440ab2b0 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:38.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57440ad190 con 0x7f57440ab2b0 2023-10-30T17:32:38.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57440abf30 con 0x7f57440aaed0 2023-10-30T17:32:38.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57440aba40 con 0x7f57440a65a0 2023-10-30T17:32:38.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7f5749483640 10 monclient(hunting): authenticate will time out at 1341.242712s 2023-10-30T17:32:38.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-10-30T17:32:38.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-10-30T17:32:38.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7facd8034da0 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:38.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd80c9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 10 monclient(hunting): picked mon.noname-b con 0x7facd80bfc20 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facd8035180 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:38.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd802d080 con 0x7facd8034da0 2023-10-30T17:32:38.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd802d350 con 0x7facd8035180 2023-10-30T17:32:38.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd80c3650 con 0x7facd80bfc20 2023-10-30T17:32:38.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.944 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.919+0000 7face4cf2640 10 monclient(hunting): authenticate will time out at 1341.243013s 2023-10-30T17:32:38.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd680a8720 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:38.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efd680a8b00 0x7efd681454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 10 monclient(hunting): picked mon.noname-c con 0x7efd680a8b00 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:38.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd68145a20 0x7efd6814c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 10 monclient(hunting): picked mon.noname-a con 0x7efd68145a20 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:38.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd680a9fd0 con 0x7efd680a8720 2023-10-30T17:32:38.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd680aa240 con 0x7efd68145a20 2023-10-30T17:32:38.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd680aa520 con 0x7efd680a8b00 2023-10-30T17:32:38.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7efd6f6d7640 10 monclient(hunting): authenticate will time out at 1341.243519s 2023-10-30T17:32:38.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7f5742ffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f57440aaed0 0x7f57440a9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.920+0000 7f57427fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 0x7f57440a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7facddd9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd80c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7f5742ffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f57440aaed0 0x7f57440a9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:39894/0 (socket says 172.21.15.170:39894) 2023-10-30T17:32:38.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7f5742ffd640 1 -- 172.21.15.170:0/2864919862 learned_addr learned my addr 172.21.15.170:0/2864919862 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7f5742ffd640 10 monclient(hunting): get_auth_request con 0x7f57440aaed0 auth_method 0 2023-10-30T17:32:38.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7f5742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7f5742ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7f5742ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7efd6cc72640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efd680a8b00 0x7efd681454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7facded9d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7facd8035180 0x7facd80d03b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:39552/0 (socket says 172.21.15.170:39552) 2023-10-30T17:32:38.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7facded9d640 1 -- 172.21.15.170:0/1526360390 learned_addr learned my addr 172.21.15.170:0/1526360390 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:38.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7f57427fc640 10 monclient(hunting): get_auth_request con 0x7f57440a65a0 auth_method 0 2023-10-30T17:32:38.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7f57427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7f57427fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7f57427fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7efd6d473640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7efd680a8720 0x7efd6813ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:51476/0 (socket says 172.21.15.170:51476) 2023-10-30T17:32:38.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7efd6d473640 1 -- 172.21.15.170:0/2330099276 learned_addr learned my addr 172.21.15.170:0/2330099276 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:38.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7efd6cc72640 10 monclient(hunting): get_auth_request con 0x7efd680a8b00 auth_method 0 2023-10-30T17:32:38.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7efd6cc72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7efd6cc72640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.921+0000 7efd6cc72640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.922+0000 7ff9e4a21640 1 Processor -- start 2023-10-30T17:32:38.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.922+0000 7ff9e4a21640 1 -- start start 2023-10-30T17:32:38.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.922+0000 7ff9e4a21640 10 monclient: init 2023-10-30T17:32:38.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.922+0000 7f57427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.922+0000 7f57427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.922+0000 7f57427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.922+0000 7f7788851640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.922+0000 7f7788851640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.922+0000 7f7788851640 10 monclient: monmap: 2023-10-30T17:32:38.957 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.957 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.957 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.923896+0000 2023-10-30T17:32:38.957 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.923896+0000 2023-10-30T17:32:38.958 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.958 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.958 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.958 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.958 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.959 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.959 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.959 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.959 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7efd6cc72640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T17:32:38.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff9e00b36a0 0x7ff9e00abd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 10 monclient(hunting): picked mon.noname-a con 0x7ff9e00b36a0 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:38.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff9e00ac920 0x7ff9e00a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 10 monclient(hunting): picked mon.noname-c con 0x7ff9e00ac920 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:38.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9e00a91d0 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:38.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e00ab690 con 0x7ff9e00a91d0 2023-10-30T17:32:38.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e00b3a80 con 0x7ff9e00b36a0 2023-10-30T17:32:38.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e00ac400 con 0x7ff9e00ac920 2023-10-30T17:32:38.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7ff9e4a21640 10 monclient(hunting): authenticate will time out at 1341.246177s 2023-10-30T17:32:38.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7efd6cc72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7efd6cc72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7f7788851640 1 Processor -- start 2023-10-30T17:32:38.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7f7788851640 1 -- start start 2023-10-30T17:32:38.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.923+0000 7f7788851640 10 monclient: init 2023-10-30T17:32:38.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.924+0000 7f6086da5640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.924+0000 7f6086da5640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.924+0000 7f6086da5640 10 monclient: monmap: 2023-10-30T17:32:38.966 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.966 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.966 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.925130+0000 2023-10-30T17:32:38.966 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.925130+0000 2023-10-30T17:32:38.967 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.967 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.967 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:38.967 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:38.967 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:38.968 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:38.968 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:38.968 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:38.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.924+0000 7f5742ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.924+0000 7f5742ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.924+0000 7f5742ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f57427fc640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2023-10-30T17:32:38.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f57427fc640 1 -- 172.21.15.170:0/2864919862 >> v2:172.21.15.170:3300/0 conn(0x7f57440aaed0 msgr2=0x7f57440a9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f57427fc640 1 --2- 172.21.15.170:0/2864919862 >> v2:172.21.15.170:3300/0 conn(0x7f57440aaed0 0x7f57440a9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-10-30T17:32:38.972 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f57427fc640 1 -- 172.21.15.170:0/2864919862 >> v1:172.21.15.170:6789/0 conn(0x7f57440ab2b0 legacy=0x7f574413db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.972 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f57427fc640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.972 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f57427fc640 1 monclient: found mon.noname-b 2023-10-30T17:32:38.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f6086da5640 1 Processor -- start 2023-10-30T17:32:38.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f57427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f57427fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f57427fc640 1 -- 172.21.15.170:0/2864919862 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57440ad8b0 con 0x7f57440a65a0 2023-10-30T17:32:38.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77840b7c90 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:38.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f778413eb60 0x7f778413efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 10 monclient(hunting): picked mon.noname-a con 0x7f778413eb60 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:38.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 0x7f7784145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 10 monclient(hunting): picked mon.noname-b con 0x7f7784144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77840aa740 con 0x7f77840b7c90 2023-10-30T17:32:38.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77840a9500 con 0x7f778413eb60 2023-10-30T17:32:38.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77840b8070 con 0x7f7784144f50 2023-10-30T17:32:38.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f7788851640 10 monclient(hunting): authenticate will time out at 1341.248951s 2023-10-30T17:32:38.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f6086da5640 1 -- start start 2023-10-30T17:32:38.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.925+0000 7f6086da5640 10 monclient: init 2023-10-30T17:32:38.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:38.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-10-30T17:32:38.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60800b7c90 0x7f60800a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 10 monclient(hunting): picked mon.noname-b con 0x7f60800b7c90 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:38.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f608013eb60 0x7f608013ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 10 monclient(hunting): picked mon.noname-c con 0x7f608013eb60 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:38.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6080144f50 0x7f60801453c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:38.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 10 monclient(hunting): picked mon.noname-a con 0x7f6080144f50 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:38.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60800a9500 con 0x7f6080144f50 2023-10-30T17:32:38.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60800b8070 con 0x7f608013eb60 2023-10-30T17:32:38.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:38.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60800aa740 con 0x7f60800b7c90 2023-10-30T17:32:38.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:38.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7f6086da5640 10 monclient(hunting): authenticate will time out at 1341.249401s 2023-10-30T17:32:38.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7efd6cc72640 10 monclient(hunting): handle_auth_done global_id 4318 payload 274 2023-10-30T17:32:38.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7efd6cc72640 1 -- 172.21.15.170:0/2330099276 >> v2:172.21.15.170:3300/0 conn(0x7efd68145a20 msgr2=0x7efd6814c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7efd6cc72640 1 --2- 172.21.15.170:0/2330099276 >> v2:172.21.15.170:3300/0 conn(0x7efd68145a20 0x7efd6814c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:38.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7efd6cc72640 1 -- 172.21.15.170:0/2330099276 >> v1:172.21.15.170:6789/0 conn(0x7efd680a8720 legacy=0x7efd6813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7efd6cc72640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7efd6cc72640 1 monclient: found mon.noname-c 2023-10-30T17:32:38.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7efd6cc72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7efd6cc72640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:32:38.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7efd6cc72640 1 -- 172.21.15.170:0/2330099276 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6813f6a0 con 0x7efd680a8b00 2023-10-30T17:32:38.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7facddd9b640 10 monclient(hunting): get_auth_request con 0x7facd80bfc20 auth_method 0 2023-10-30T17:32:38.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7facddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7facddd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.926+0000 7facddd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.927+0000 7ff9deffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff9e00ac920 0x7ff9e00a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:38.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.927+0000 7facddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.927+0000 7facddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.927+0000 7facddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.927+0000 7facdd59a640 1 -- 172.21.15.170:0/1526360390 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462499877 0 0) 0x7facd802d350 con 0x7facd8035180 2023-10-30T17:32:38.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.927+0000 7facdd59a640 10 monclient(hunting): _negotiate 2023-10-30T17:32:38.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.927+0000 7facdd59a640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.927+0000 7facdd59a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.927+0000 7facdd59a640 10 monclient(hunting): my global_id is 4321 2023-10-30T17:32:38.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.927+0000 7facdd59a640 1 -- 172.21.15.170:0/1526360390 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc8003520 con 0x7facd8035180 2023-10-30T17:32:38.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.928+0000 7f778259c640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f77840b7c90 0x7f77840a8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:39562/0 (socket says 172.21.15.170:39562) 2023-10-30T17:32:38.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.928+0000 7f778259c640 1 -- 172.21.15.170:0/2610760268 learned_addr learned my addr 172.21.15.170:0/2610760268 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:38.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.928+0000 7ff9deffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff9e00ac920 0x7ff9e00a9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60568/0 (socket says 172.21.15.170:60568) 2023-10-30T17:32:38.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.928+0000 7ff9deffd640 1 -- 172.21.15.170:0/4046080084 learned_addr learned my addr 172.21.15.170:0/4046080084 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:38.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.928+0000 7ff9deffd640 10 monclient(hunting): get_auth_request con 0x7ff9e00ac920 auth_method 0 2023-10-30T17:32:38.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.928+0000 7ff9deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:38.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.928+0000 7ff9deffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:38.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.928+0000 7ff9deffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:38.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7facddd9b640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2023-10-30T17:32:38.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7facddd9b640 1 -- 172.21.15.170:0/1526360390 >> v1:172.21.15.177:6789/0 conn(0x7facd8035180 legacy=0x7facd80d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:38.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7facddd9b640 1 -- 172.21.15.170:0/1526360390 >> v1:172.21.15.170:6789/0 conn(0x7facd8034da0 legacy=0x7facd80bf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:38.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7facddd9b640 10 monclient: _finish_hunting 0 2023-10-30T17:32:38.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7facddd9b640 1 monclient: found mon.noname-b 2023-10-30T17:32:38.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7facddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:38.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7facddd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:38.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7facddd9b640 1 -- 172.21.15.170:0/1526360390 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd802d870 con 0x7facd80bfc20 2023-10-30T17:32:38.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:38.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:38.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:38.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7f58fbb41640 10 monclient: get_monmap_and_config 2023-10-30T17:32:38.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7f58fbb41640 10 monclient: build_initial_monmap 2023-10-30T17:32:38.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7f58fbb41640 10 monclient: monmap: 2023-10-30T17:32:38.998 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:38.998 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:38.999 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.930339+0000 2023-10-30T17:32:38.999 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.930339+0000 2023-10-30T17:32:38.999 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:38.999 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:38.999 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:39.000 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:39.000 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:39.000 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:39.000 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:39.000 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7fc8925c9640 10 monclient: get_monmap_and_config 2023-10-30T17:32:39.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7fc8925c9640 10 monclient: build_initial_monmap 2023-10-30T17:32:39.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.929+0000 7fc8925c9640 10 monclient: monmap: 2023-10-30T17:32:39.001 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:39.002 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.002 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:38.930676+0000 2023-10-30T17:32:39.002 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:38.930676+0000 2023-10-30T17:32:39.002 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:39.002 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.003 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:39.003 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:39.003 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:39.003 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:39.003 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:39.004 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7f58fbb41640 1 Processor -- start 2023-10-30T17:32:39.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7f58fbb41640 1 -- start start 2023-10-30T17:32:39.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7f58fbb41640 10 monclient: init 2023-10-30T17:32:39.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7fc8925c9640 1 Processor -- start 2023-10-30T17:32:39.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7fc8925c9640 1 -- start start 2023-10-30T17:32:39.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7fc8925c9640 10 monclient: init 2023-10-30T17:32:39.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7f58fbb41640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:39.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7f58fbb41640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-10-30T17:32:39.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7f58fbb41640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f58f40a8720 0x7f58f413ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7f58fbb41640 10 monclient(hunting): picked mon.noname-c con 0x7f58f40a8720 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:39.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7f58fbb41640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f413f4d0 0x7f58f41454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7f58fbb41640 10 monclient(hunting): picked mon.noname-b con 0x7f58f413f4d0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:39.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7f58fbb41640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58f40a8b00 0x7f58f4145a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7f58fbb41640 10 monclient(hunting): picked mon.noname-a con 0x7f58f40a8b00 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:39.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7f58fbb41640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7f58fbb41640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f4145f60 con 0x7f58f40a8b00 2023-10-30T17:32:39.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7f58fbb41640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7f58fbb41640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f40a9fd0 con 0x7f58f40a8720 2023-10-30T17:32:39.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7f58fbb41640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7f58fbb41640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40aa240 con 0x7f58f413f4d0 2023-10-30T17:32:39.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7f58fbb41640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:39.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7f58fbb41640 10 monclient(hunting): authenticate will time out at 1341.254118s 2023-10-30T17:32:39.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.930+0000 7ff9deffd640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2023-10-30T17:32:39.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7ff9deffd640 1 -- 172.21.15.170:0/4046080084 >> v2:172.21.15.170:3300/0 conn(0x7ff9e00b36a0 msgr2=0x7ff9e00abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:39.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7ff9deffd640 1 --2- 172.21.15.170:0/4046080084 >> v2:172.21.15.170:3300/0 conn(0x7ff9e00b36a0 0x7ff9e00abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7ff9deffd640 1 -- 172.21.15.170:0/4046080084 >> v1:172.21.15.170:6789/0 conn(0x7ff9e00a91d0 legacy=0x7ff9e013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:39.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7ff9deffd640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7ff9deffd640 1 monclient: found mon.noname-c 2023-10-30T17:32:39.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7ff9deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7ff9deffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7ff9deffd640 1 -- 172.21.15.170:0/4046080084 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e013b0c0 con 0x7ff9e00ac920 2023-10-30T17:32:39.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:39.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-10-30T17:32:39.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc88c0b7c90 0x7fc88c0a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 10 monclient(hunting): picked mon.noname-a con 0x7fc88c0b7c90 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:39.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc88c13eb60 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:39.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc88c144f50 0x7fc88c145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc88c144f50 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:39.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88c0aa740 con 0x7fc88c13eb60 2023-10-30T17:32:39.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c0a9500 con 0x7fc88c0b7c90 2023-10-30T17:32:39.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c13f6c0 con 0x7fc88c144f50 2023-10-30T17:32:39.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:39.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.931+0000 7fc8925c9640 10 monclient(hunting): authenticate will time out at 1341.255058s 2023-10-30T17:32:39.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.932+0000 7f57427fc640 1 --2- 172.21.15.170:0/2864919862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 0x7f57440a9760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f57340097a0 tx=0x7f573402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e18ddb6aca6857d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.932+0000 7f607ffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f608013eb60 0x7f608013ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.932+0000 7f607ffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f608013eb60 0x7f608013ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60582/0 (socket says 172.21.15.170:60582) 2023-10-30T17:32:39.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.932+0000 7f607ffff640 1 -- 172.21.15.170:0/896178659 learned_addr learned my addr 172.21.15.170:0/896178659 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.932+0000 7f607ffff640 10 monclient(hunting): get_auth_request con 0x7f608013eb60 auth_method 0 2023-10-30T17:32:39.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.932+0000 7f607ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.932+0000 7f607ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.932+0000 7f607ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.933+0000 7facddd9b640 1 --2- 172.21.15.170:0/1526360390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd80c9020 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7facd00097d0 tx=0x7facd002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb57027bad6ef94 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.936+0000 7f607ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.936+0000 7f607ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.936+0000 7f607ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.936+0000 7f778159a640 1 -- 172.21.15.170:0/2610760268 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798436233 0 0) 0x7f77840aa740 con 0x7f77840b7c90 2023-10-30T17:32:39.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.936+0000 7f778159a640 10 monclient(hunting): _negotiate 2023-10-30T17:32:39.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.936+0000 7f778159a640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.936+0000 7f778159a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.936+0000 7f778159a640 10 monclient(hunting): my global_id is 4327 2023-10-30T17:32:39.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.936+0000 7f778159a640 1 -- 172.21.15.170:0/2610760268 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f776c003520 con 0x7f77840b7c90 2023-10-30T17:32:39.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.937+0000 7ff9deffd640 1 --2- 172.21.15.170:0/4046080084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00ac920 0x7ff9e00a9b70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00b3cf0 tx=0x7ff9d002c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59ea2a315148e9a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.937+0000 7f7781d9b640 1 --2- 172.21.15.170:0/2610760268 >> v2:172.21.15.170:3300/0 conn(0x7f778413eb60 0x7f778413efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.937+0000 7efd6cc72640 1 --2- 172.21.15.170:0/2330099276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd680a8b00 0x7efd681454e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efd680673a0 tx=0x7efd5802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d462cfd3f4ecf9a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.938+0000 7f7781d9b640 10 monclient(hunting): get_auth_request con 0x7f778413eb60 auth_method 0 2023-10-30T17:32:39.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.938+0000 7f7781d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.938+0000 7f7781d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.938+0000 7f7781d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f7782d9d640 1 --2- 172.21.15.170:0/2610760268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 0x7f7784145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f607ffff640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2023-10-30T17:32:39.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f607ffff640 1 -- 172.21.15.170:0/896178659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60800b7c90 msgr2=0x7f60800a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:39.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f607ffff640 1 --2- 172.21.15.170:0/896178659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60800b7c90 0x7f60800a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f607ffff640 1 -- 172.21.15.170:0/896178659 >> v2:172.21.15.170:3300/0 conn(0x7f6080144f50 msgr2=0x7f60801453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:39.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f607ffff640 1 --2- 172.21.15.170:0/896178659 >> v2:172.21.15.170:3300/0 conn(0x7f6080144f50 0x7f60801453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f607ffff640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f607ffff640 1 monclient: found mon.noname-c 2023-10-30T17:32:39.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f607ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f607ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f607ffff640 1 -- 172.21.15.170:0/896178659 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60800aa9b0 con 0x7f608013eb60 2023-10-30T17:32:39.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f7781d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f7781d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f7781d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.939+0000 7f58fa0de640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58f40a8b00 0x7f58f4145a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f7782d9d640 10 monclient(hunting): get_auth_request con 0x7f7784144f50 auth_method 0 2023-10-30T17:32:39.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f7782d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f7782d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f7782d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7fc88bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc88c0b7c90 0x7fc88c0a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f58fa0de640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58f40a8b00 0x7f58f4145a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:39922/0 (socket says 172.21.15.170:39922) 2023-10-30T17:32:39.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f58fa0de640 1 -- 172.21.15.170:0/3872762520 learned_addr learned my addr 172.21.15.170:0/3872762520 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f58fa0de640 10 monclient(hunting): get_auth_request con 0x7f58f40a8b00 auth_method 0 2023-10-30T17:32:39.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f58fa0de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f58fa0de640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f58fa0de640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 1 -- 172.21.15.170:0/2864919862 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734031830 con 0x7f57440a65a0 2023-10-30T17:32:39.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:39.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 10 monclient: dump: 2023-10-30T17:32:39.031 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.031 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.032 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.032 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.032 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.032 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.032 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.032 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.033 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.033 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 1 -- 172.21.15.170:0/2864919862 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5734031990 con 0x7f57440a65a0 2023-10-30T17:32:39.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5749483640 5 monclient: authenticate success, global_id 4286 2023-10-30T17:32:39.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5749483640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:39.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5749483640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:39.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5749483640 4 set_mon_vals no callback set 2023-10-30T17:32:39.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 1 -- 172.21.15.170:0/2864919862 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734031d20 con 0x7f57440a65a0 2023-10-30T17:32:39.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:39.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.940+0000 7f5741ffb640 10 monclient: dump: 2023-10-30T17:32:39.035 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.036 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.036 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.036 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.036 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.036 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.037 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.037 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.037 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.037 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f7782d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f7782d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f7782d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f58f90dc640 1 --2- 172.21.15.170:0/3872762520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f413f4d0 0x7f58f41454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.039 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f5749483640 10 monclient: shutdown 2023-10-30T17:32:39.039 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f5749483640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f5749483640 1 -- 172.21.15.170:0/2864919862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 msgr2=0x7f57440a9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f5749483640 1 --2- 172.21.15.170:0/2864919862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 0x7f57440a9760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f57340097a0 tx=0x7f573402b8d0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f5749483640 1 -- 172.21.15.170:0/2864919862 shutdown_connections 2023-10-30T17:32:39.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f5749483640 1 --2- 172.21.15.170:0/2864919862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 0x7f57440a9760 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f5749483640 1 --2- 172.21.15.170:0/2864919862 >> v2:172.21.15.170:3300/0 conn(0x7f57440aaed0 0x7f57440a9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f5749483640 1 -- 172.21.15.170:0/2864919862 >> 172.21.15.170:0/2864919862 conn(0x7f57440b3110 msgr2=0x7f57440b3520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.941+0000 7f5749483640 1 -- 172.21.15.170:0/2864919862 shutdown_connections 2023-10-30T17:32:39.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.942+0000 7f5749483640 1 -- 172.21.15.170:0/2864919862 wait complete. 2023-10-30T17:32:39.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.942+0000 7fc88bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc88c0b7c90 0x7fc88c0a8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:39938/0 (socket says 172.21.15.170:39938) 2023-10-30T17:32:39.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.942+0000 7fc88bfff640 1 -- 172.21.15.170:0/3951768484 learned_addr learned my addr 172.21.15.170:0/3951768484 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.942+0000 7fc88bfff640 10 monclient(hunting): get_auth_request con 0x7fc88c0b7c90 auth_method 0 2023-10-30T17:32:39.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.942+0000 7fc88bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.942+0000 7fc88bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.942+0000 7f5749483640 10 monclient: build_initial_monmap 2023-10-30T17:32:39.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.942+0000 7f5749483640 10 monclient: monmap: 2023-10-30T17:32:39.043 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:39.043 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:39.044 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:39.044 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:39.044 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:39.044 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.044 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:39.045 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:39.045 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:39.045 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.045 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.943+0000 7f58fa0de640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.046 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.943+0000 7f58fa0de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.046 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.943+0000 7f58fa0de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.046 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.942+0000 7fc88bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.943+0000 7f5749483640 1 librados: starting msgr at 2023-10-30T17:32:39.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.943+0000 7f5749483640 1 librados: starting objecter 2023-10-30T17:32:39.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.943+0000 7f5749483640 1 Processor -- start 2023-10-30T17:32:39.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.943+0000 7f5749483640 1 -- start start 2023-10-30T17:32:39.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.943+0000 7f5749483640 1 librados: setting wanted keys 2023-10-30T17:32:39.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.943+0000 7f5749483640 1 librados: calling monclient init 2023-10-30T17:32:39.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.943+0000 7f5749483640 10 monclient: init 2023-10-30T17:32:39.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f7782d9d640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2023-10-30T17:32:39.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f7782d9d640 1 -- 172.21.15.170:0/2610760268 >> v2:172.21.15.170:3300/0 conn(0x7f778413eb60 msgr2=0x7f778413efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:39.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f7782d9d640 1 --2- 172.21.15.170:0/2610760268 >> v2:172.21.15.170:3300/0 conn(0x7f778413eb60 0x7f778413efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f7782d9d640 1 -- 172.21.15.170:0/2610760268 >> v1:172.21.15.177:6789/0 conn(0x7f77840b7c90 legacy=0x7f77840a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f7782d9d640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f7782d9d640 1 monclient: found mon.noname-b 2023-10-30T17:32:39.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 1 -- 172.21.15.170:0/4046080084 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0008df0 con 0x7ff9e00ac920 2023-10-30T17:32:39.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:39.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 10 monclient: dump: 2023-10-30T17:32:39.051 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.051 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.052 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.052 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.052 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.052 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.052 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.053 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.053 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.053 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 1 -- 172.21.15.170:0/4046080084 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d0004030 con 0x7ff9e00ac920 2023-10-30T17:32:39.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 1 -- 172.21.15.170:0/4046080084 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0004420 con 0x7ff9e00ac920 2023-10-30T17:32:39.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:39.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9de7fc640 10 monclient: dump: 2023-10-30T17:32:39.055 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.055 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.055 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.055 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.056 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.056 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.056 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.056 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.056 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.057 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:39.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:32:39.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 0x7f57441300f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 10 monclient(hunting): picked mon.noname-c con 0x7f57440a65a0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:39.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57440aaed0 0x7f574413f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 10 monclient(hunting): picked mon.noname-a con 0x7f57440aaed0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:39.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57440ab2b0 0x7f574406d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 10 monclient(hunting): picked mon.noname-b con 0x7f57440ab2b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:39.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5744140440 con 0x7f57440aaed0 2023-10-30T17:32:39.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574406db70 con 0x7f57440ab2b0 2023-10-30T17:32:39.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5744141d70 con 0x7f57440a65a0 2023-10-30T17:32:39.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:39.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f5749483640 10 monclient(hunting): authenticate will time out at 1341.267647s 2023-10-30T17:32:39.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f7782d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f7782d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 1 -- 172.21.15.170:0/1526360390 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0031800 con 0x7facd80bfc20 2023-10-30T17:32:39.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:39.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 10 monclient: dump: 2023-10-30T17:32:39.063 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.063 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.063 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.063 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.063 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.064 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.064 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.064 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.064 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.064 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 1 -- 172.21.15.170:0/1526360390 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd0031960 con 0x7facd80bfc20 2023-10-30T17:32:39.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 1 -- 172.21.15.170:0/1526360390 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0031cf0 con 0x7facd80bfc20 2023-10-30T17:32:39.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:39.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7facdd59a640 10 monclient: dump: 2023-10-30T17:32:39.066 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.066 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.067 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.067 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.067 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.067 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.067 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.068 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.068 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.068 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7face4cf2640 5 monclient: authenticate success, global_id 4289 2023-10-30T17:32:39.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7face4cf2640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:39.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7face4cf2640 4 set_mon_vals no callback set 2023-10-30T17:32:39.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7f7782d9d640 1 -- 172.21.15.170:0/2610760268 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77840aa9b0 con 0x7f7784144f50 2023-10-30T17:32:39.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9e4a21640 5 monclient: authenticate success, global_id 4324 2023-10-30T17:32:39.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9e4a21640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:39.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.944+0000 7ff9e4a21640 4 set_mon_vals no callback set 2023-10-30T17:32:39.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7f58fa0de640 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2023-10-30T17:32:39.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7f58fa0de640 1 -- 172.21.15.170:0/3872762520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f413f4d0 msgr2=0x7f58f41454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:39.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7f58fa0de640 1 --2- 172.21.15.170:0/3872762520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f413f4d0 0x7f58f41454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7f58fa0de640 1 -- 172.21.15.170:0/3872762520 >> v2:172.21.15.177:3300/0 conn(0x7f58f40a8720 msgr2=0x7f58f413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:39.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7f58fa0de640 1 --2- 172.21.15.170:0/3872762520 >> v2:172.21.15.177:3300/0 conn(0x7f58f40a8720 0x7f58f413ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7f58fa0de640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7f58fa0de640 1 monclient: found mon.noname-a 2023-10-30T17:32:39.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7f58fa0de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7f58fa0de640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7f58fa0de640 1 -- 172.21.15.170:0/3872762520 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f40aa520 con 0x7f58f40a8b00 2023-10-30T17:32:39.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7fc890b66640 1 --2- 172.21.15.170:0/3951768484 >> v2:172.21.15.177:3300/0 conn(0x7fc88c144f50 0x7fc88c145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7fc88bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7fc88bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7fc88bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7face4cf2640 10 monclient: shutdown 2023-10-30T17:32:39.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7face4cf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7face4cf2640 1 -- 172.21.15.170:0/1526360390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 msgr2=0x7facd80c9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7face4cf2640 1 --2- 172.21.15.170:0/1526360390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd80c9020 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7facd00097d0 tx=0x7facd002b8a0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7face4cf2640 1 -- 172.21.15.170:0/1526360390 shutdown_connections 2023-10-30T17:32:39.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.945+0000 7face4cf2640 1 --2- 172.21.15.170:0/1526360390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd80c9020 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7face4cf2640 1 -- 172.21.15.170:0/1526360390 >> 172.21.15.170:0/1526360390 conn(0x7facd80061f0 msgr2=0x7facd8006600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7face4cf2640 1 -- 172.21.15.170:0/1526360390 shutdown_connections 2023-10-30T17:32:39.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7ff9e4a21640 10 monclient: shutdown 2023-10-30T17:32:39.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7ff9e4a21640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7ff9e4a21640 1 -- 172.21.15.170:0/4046080084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00ac920 msgr2=0x7ff9e00a9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7ff9e4a21640 1 --2- 172.21.15.170:0/4046080084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00ac920 0x7ff9e00a9b70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00b3cf0 tx=0x7ff9d002c980 comp rx=0 tx=0).stop 2023-10-30T17:32:39.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7fc890b66640 10 monclient(hunting): get_auth_request con 0x7fc88c144f50 auth_method 0 2023-10-30T17:32:39.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7fc890b66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7fc890b66640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7fc890b66640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7ff9e4a21640 1 -- 172.21.15.170:0/4046080084 shutdown_connections 2023-10-30T17:32:39.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7ff9e4a21640 1 --2- 172.21.15.170:0/4046080084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00ac920 0x7ff9e00a9b70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7ff9e4a21640 1 --2- 172.21.15.170:0/4046080084 >> v2:172.21.15.170:3300/0 conn(0x7ff9e00b36a0 0x7ff9e00abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7ff9e4a21640 1 -- 172.21.15.170:0/4046080084 >> 172.21.15.170:0/4046080084 conn(0x7ff9e00afdf0 msgr2=0x7ff9e00b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7ff9e4a21640 1 -- 172.21.15.170:0/4046080084 shutdown_connections 2023-10-30T17:32:39.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7ff9e4a21640 1 -- 172.21.15.170:0/4046080084 wait complete. 2023-10-30T17:32:39.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.946+0000 7face4cf2640 1 -- 172.21.15.170:0/1526360390 wait complete. 2023-10-30T17:32:39.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7ff9e4a21640 10 monclient: build_initial_monmap 2023-10-30T17:32:39.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7ff9e4a21640 10 monclient: monmap: 2023-10-30T17:32:39.082 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:39.083 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:39.083 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:39.083 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:39.083 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:39.083 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.083 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:39.084 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:39.084 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:39.084 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7face4cf2640 10 monclient: build_initial_monmap 2023-10-30T17:32:39.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7face4cf2640 10 monclient: monmap: 2023-10-30T17:32:39.085 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:39.085 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:39.085 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:39.085 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:39.086 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:39.086 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.086 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:39.086 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:39.086 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:39.086 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7f7782d9d640 1 --2- 172.21.15.170:0/2610760268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 0x7f7784145400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f777c009d90 tx=0x7f777c02be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67a3052060c08dcc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7f607ffff640 1 --2- 172.21.15.170:0/896178659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f608013eb60 0x7f608013ef80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f60800b82b0 tx=0x7f606c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1502b6f0f6668bde server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7ff9e4a21640 1 librados: starting msgr at 2023-10-30T17:32:39.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7ff9e4a21640 1 librados: starting objecter 2023-10-30T17:32:39.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7face4cf2640 1 librados: starting msgr at 2023-10-30T17:32:39.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7face4cf2640 1 librados: starting objecter 2023-10-30T17:32:39.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7ff9e4a21640 1 Processor -- start 2023-10-30T17:32:39.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.947+0000 7face4cf2640 1 Processor -- start 2023-10-30T17:32:39.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7face4cf2640 1 -- start start 2023-10-30T17:32:39.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7face4cf2640 1 librados: setting wanted keys 2023-10-30T17:32:39.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7face4cf2640 1 librados: calling monclient init 2023-10-30T17:32:39.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7face4cf2640 10 monclient: init 2023-10-30T17:32:39.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 1 -- start start 2023-10-30T17:32:39.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 1 librados: setting wanted keys 2023-10-30T17:32:39.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 1 librados: calling monclient init 2023-10-30T17:32:39.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 10 monclient: init 2023-10-30T17:32:39.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7face4cf2640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:39.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7face4cf2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:32:39.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7face4cf2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facd8034da0 0x7facd80c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 10 monclient(hunting): picked mon.noname-a con 0x7facd8034da0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:39.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7fc88bfff640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2023-10-30T17:32:39.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:39.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:32:39.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9e00a91d0 0x7ff9e0134030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 10 monclient(hunting): picked mon.noname-a con 0x7ff9e00a91d0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:39.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7fc88bfff640 1 -- 172.21.15.170:0/3951768484 >> v2:172.21.15.177:3300/0 conn(0x7fc88c144f50 msgr2=0x7fc88c145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:39.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9e00ac920 0x7ff9e013a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 10 monclient(hunting): picked mon.noname-c con 0x7ff9e00ac920 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:39.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00b36a0 0x7ff9e00adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 10 monclient(hunting): picked mon.noname-b con 0x7ff9e00b36a0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:39.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.948+0000 7ff9e4a21640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00af150 con 0x7ff9e00a91d0 2023-10-30T17:32:39.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7ff9e4a21640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7ff9e4a21640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00af330 con 0x7ff9e00b36a0 2023-10-30T17:32:39.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7ff9e4a21640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7ff9e4a21640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00ad9a0 con 0x7ff9e00ac920 2023-10-30T17:32:39.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7ff9e4a21640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:39.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7fc88bfff640 1 --2- 172.21.15.170:0/3951768484 >> v2:172.21.15.177:3300/0 conn(0x7fc88c144f50 0x7fc88c145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7ff9e4a21640 10 monclient(hunting): authenticate will time out at 1341.272081s 2023-10-30T17:32:39.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7fc88bfff640 1 -- 172.21.15.170:0/3951768484 >> v1:172.21.15.170:6789/0 conn(0x7fc88c13eb60 legacy=0x7fc88c13efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:39.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7fc88bfff640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7fc88bfff640 1 monclient: found mon.noname-a 2023-10-30T17:32:39.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7fc88bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7fc88bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7fc88bfff640 1 -- 172.21.15.170:0/3951768484 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c0aa9b0 con 0x7fc88c0b7c90 2023-10-30T17:32:39.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd8035180 0x7facd815d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 10 monclient(hunting): picked mon.noname-b con 0x7facd8035180 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:39.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd815d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 10 monclient(hunting): picked mon.noname-c con 0x7facd80bfc20 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:39.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd815a7e0 con 0x7facd8034da0 2023-10-30T17:32:39.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd80c3910 con 0x7facd8035180 2023-10-30T17:32:39.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd80c2b80 con 0x7facd80bfc20 2023-10-30T17:32:39.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:39.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.949+0000 7face4cf2640 10 monclient(hunting): authenticate will time out at 1341.272786s 2023-10-30T17:32:39.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.951+0000 7f57427fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57440aaed0 0x7f574413f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 1 -- 172.21.15.170:0/2330099276 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58031830 con 0x7efd680a8b00 2023-10-30T17:32:39.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:39.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 10 monclient: dump: 2023-10-30T17:32:39.105 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.105 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.105 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.105 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.105 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.106 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.106 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.106 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.106 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.106 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 1 -- 172.21.15.170:0/2330099276 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd58031990 con 0x7efd680a8b00 2023-10-30T17:32:39.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd6f6d7640 5 monclient: authenticate success, global_id 4318 2023-10-30T17:32:39.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 1 -- 172.21.15.170:0/2330099276 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58031d20 con 0x7efd680a8b00 2023-10-30T17:32:39.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7f57427fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57440aaed0 0x7f574413f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:39946/0 (socket says 172.21.15.170:39946) 2023-10-30T17:32:39.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7f57427fc640 1 -- 172.21.15.170:0/3889062015 learned_addr learned my addr 172.21.15.170:0/3889062015 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:39.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd5ffff640 10 monclient: dump: 2023-10-30T17:32:39.109 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.109 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.110 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.110 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.110 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.110 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.110 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.110 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.111 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.111 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd6f6d7640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:39.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7efd6f6d7640 4 set_mon_vals no callback set 2023-10-30T17:32:39.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7f57427fc640 10 monclient(hunting): get_auth_request con 0x7f57440aaed0 auth_method 0 2023-10-30T17:32:39.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7f57427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7f57427fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7f57427fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7f57437fe640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57440ab2b0 0x7f574406d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.953+0000 7f58fa0de640 1 --2- 172.21.15.170:0/3872762520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58f40a8b00 0x7f58f4145a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f58f4067150 tx=0x7f58ec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2603d34af23b1ef9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.114 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7efd6f6d7640 10 monclient: shutdown 2023-10-30T17:32:39.114 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7efd6f6d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.114 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7efd6f6d7640 1 -- 172.21.15.170:0/2330099276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd680a8b00 msgr2=0x7efd681454e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.114 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7efd6f6d7640 1 --2- 172.21.15.170:0/2330099276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd680a8b00 0x7efd681454e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efd680673a0 tx=0x7efd5802b8d0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7ff9dffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00b36a0 0x7ff9e00adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7efd6f6d7640 1 -- 172.21.15.170:0/2330099276 shutdown_connections 2023-10-30T17:32:39.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7efd6f6d7640 1 --2- 172.21.15.170:0/2330099276 >> v2:172.21.15.170:3300/0 conn(0x7efd68145a20 0x7efd6814c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7efd6f6d7640 1 --2- 172.21.15.170:0/2330099276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd680a8b00 0x7efd681454e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7efd6f6d7640 1 -- 172.21.15.170:0/2330099276 >> 172.21.15.170:0/2330099276 conn(0x7efd680b7300 msgr2=0x7efd680b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7efd6f6d7640 1 -- 172.21.15.170:0/2330099276 shutdown_connections 2023-10-30T17:32:39.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7efd6f6d7640 1 -- 172.21.15.170:0/2330099276 wait complete. 2023-10-30T17:32:39.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7f57427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.954+0000 7f57427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f57427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7ff9dffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00b36a0 0x7ff9e00adec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60618/0 (socket says 172.21.15.170:60618) 2023-10-30T17:32:39.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7ff9dffff640 1 -- 172.21.15.170:0/2086724736 learned_addr learned my addr 172.21.15.170:0/2086724736 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7ff9dffff640 10 monclient(hunting): get_auth_request con 0x7ff9e00b36a0 auth_method 0 2023-10-30T17:32:39.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7ff9dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7ff9dffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7ff9dffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7efd6f6d7640 10 monclient: build_initial_monmap 2023-10-30T17:32:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7efd6f6d7640 10 monclient: monmap: 2023-10-30T17:32:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:39.121 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.121 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:39.121 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:39.121 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:39.121 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 1 -- 172.21.15.170:0/2610760268 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c02cbd0 con 0x7f7784144f50 2023-10-30T17:32:39.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:39.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 10 monclient: dump: 2023-10-30T17:32:39.123 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.123 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.123 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.123 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.124 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.124 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.124 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.124 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.124 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.124 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 1 -- 172.21.15.170:0/2610760268 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c02cd30 con 0x7f7784144f50 2023-10-30T17:32:39.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 1 -- 172.21.15.170:0/2610760268 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c02cea0 con 0x7f7784144f50 2023-10-30T17:32:39.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:39.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f778159a640 10 monclient: dump: 2023-10-30T17:32:39.126 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.126 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.127 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.127 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.127 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.127 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.127 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.128 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.128 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.128 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f7788851640 5 monclient: authenticate success, global_id 4292 2023-10-30T17:32:39.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f7788851640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:39.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7f7788851640 4 set_mon_vals no callback set 2023-10-30T17:32:39.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7efd6f6d7640 1 librados: starting msgr at 2023-10-30T17:32:39.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7efd6f6d7640 1 librados: starting objecter 2023-10-30T17:32:39.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7efd6f6d7640 1 Processor -- start 2023-10-30T17:32:39.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7fc88bfff640 1 --2- 172.21.15.170:0/3951768484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c0b7c90 0x7fc88c0a8b80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0aae00 tx=0x7fc87c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95c8aa9efb7de77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7ff9deffd640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9e00ac920 0x7ff9e013a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7efd6f6d7640 1 -- start start 2023-10-30T17:32:39.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7efd6f6d7640 1 librados: setting wanted keys 2023-10-30T17:32:39.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7efd6f6d7640 1 librados: calling monclient init 2023-10-30T17:32:39.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.955+0000 7efd6f6d7640 10 monclient: init 2023-10-30T17:32:39.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7ff9deffd640 10 monclient(hunting): get_auth_request con 0x7ff9e00ac920 auth_method 0 2023-10-30T17:32:39.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7ff9deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7ff9deffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7ff9deffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7f7788851640 10 monclient: shutdown 2023-10-30T17:32:39.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7f7788851640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7f7788851640 1 -- 172.21.15.170:0/2610760268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 msgr2=0x7f7784145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.133 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7f7788851640 1 --2- 172.21.15.170:0/2610760268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 0x7f7784145400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f777c009d90 tx=0x7f777c02be20 comp rx=0 tx=0).stop 2023-10-30T17:32:39.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:39.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:32:39.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd680a8720 0x7efd68138520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 10 monclient(hunting): picked mon.noname-a con 0x7efd680a8720 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:39.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7f7788851640 1 -- 172.21.15.170:0/2610760268 shutdown_connections 2023-10-30T17:32:39.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7f7788851640 1 --2- 172.21.15.170:0/2610760268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 0x7f7784145400 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7f7788851640 1 --2- 172.21.15.170:0/2610760268 >> v2:172.21.15.170:3300/0 conn(0x7f778413eb60 0x7f778413efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd680a8b00 0x7efd681db9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 10 monclient(hunting): picked mon.noname-c con 0x7efd680a8b00 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:39.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd68145a20 0x7efd680afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 10 monclient(hunting): picked mon.noname-b con 0x7efd68145a20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:39.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd680adb60 con 0x7efd680a8720 2023-10-30T17:32:39.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7f7788851640 1 -- 172.21.15.170:0/2610760268 >> 172.21.15.170:0/2610760268 conn(0x7f7784071da0 msgr2=0x7f77840b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd681db480 con 0x7efd68145a20 2023-10-30T17:32:39.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd680aee50 con 0x7efd680a8b00 2023-10-30T17:32:39.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:39.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7efd6f6d7640 10 monclient(hunting): authenticate will time out at 1341.279801s 2023-10-30T17:32:39.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7f7788851640 1 -- 172.21.15.170:0/2610760268 shutdown_connections 2023-10-30T17:32:39.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.956+0000 7f7788851640 1 -- 172.21.15.170:0/2610760268 wait complete. 2023-10-30T17:32:39.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.957+0000 7f7788851640 10 monclient: build_initial_monmap 2023-10-30T17:32:39.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.957+0000 7f7788851640 10 monclient: monmap: 2023-10-30T17:32:39.141 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:39.141 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:39.141 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:39.141 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:39.141 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:39.142 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.142 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:39.142 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:39.142 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:39.142 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f7788851640 1 librados: starting msgr at 2023-10-30T17:32:39.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f7788851640 1 librados: starting objecter 2023-10-30T17:32:39.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f7788851640 1 Processor -- start 2023-10-30T17:32:39.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f7788851640 1 -- start start 2023-10-30T17:32:39.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f7788851640 1 librados: setting wanted keys 2023-10-30T17:32:39.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f7788851640 1 librados: calling monclient init 2023-10-30T17:32:39.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f7788851640 10 monclient: init 2023-10-30T17:32:39.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7ff9dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7ff9dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7ff9dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f57437fe640 10 monclient(hunting): get_auth_request con 0x7f57440ab2b0 auth_method 0 2023-10-30T17:32:39.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f57437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f57437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f57437fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7ff9deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f7788851640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:39.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f7788851640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:32:39.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f7788851640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77840b7c90 0x7f7784144540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 10 monclient(hunting): picked mon.noname-b con 0x7f77840b7c90 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:39.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.958+0000 7f5742ffd640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 0x7f57441300f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f778413eb60 0x7f77841438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 10 monclient(hunting): picked mon.noname-a con 0x7f778413eb60 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:39.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 0x7f77841d65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 10 monclient(hunting): picked mon.noname-c con 0x7f7784144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:39.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7784144d00 con 0x7f778413eb60 2023-10-30T17:32:39.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7784143de0 con 0x7f77840b7c90 2023-10-30T17:32:39.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7784144090 con 0x7f7784144f50 2023-10-30T17:32:39.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:39.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f7788851640 10 monclient(hunting): authenticate will time out at 1341.282124s 2023-10-30T17:32:39.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f57427fc640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2023-10-30T17:32:39.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f57427fc640 1 -- 172.21.15.170:0/3889062015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 msgr2=0x7f57441300f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f57427fc640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 0x7f57441300f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f57427fc640 1 -- 172.21.15.170:0/3889062015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57440ab2b0 msgr2=0x7f574406d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f57427fc640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57440ab2b0 0x7f574406d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f57427fc640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f57427fc640 1 monclient: found mon.noname-a 2023-10-30T17:32:39.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f57427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f57427fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.959+0000 7f57427fc640 1 -- 172.21.15.170:0/3889062015 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5734009400 con 0x7f57440aaed0 2023-10-30T17:32:39.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.961+0000 7ff9dffff640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2023-10-30T17:32:39.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.961+0000 7ff9dffff640 1 -- 172.21.15.170:0/2086724736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9e00ac920 msgr2=0x7ff9e013a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.961+0000 7ff9dffff640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9e00ac920 0x7ff9e013a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.961+0000 7ff9dffff640 1 -- 172.21.15.170:0/2086724736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9e00a91d0 msgr2=0x7ff9e0134030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.961+0000 7ff9dffff640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9e00a91d0 0x7ff9e0134030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.961+0000 7ff9dffff640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.961+0000 7ff9dffff640 1 monclient: found mon.noname-b 2023-10-30T17:32:39.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.961+0000 7ff9dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.961+0000 7ff9dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.961+0000 7ff9dffff640 1 -- 172.21.15.170:0/2086724736 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d0007040 con 0x7ff9e00b36a0 2023-10-30T17:32:39.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.964+0000 7ff9dffff640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00b36a0 0x7ff9e00adec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c780 tx=0x7ff9d800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.965+0000 7ff9c7fff640 10 client.?.objecter ms_handle_connect 0x7ff9e00b36a0 2023-10-30T17:32:39.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.965+0000 7ff9c7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:39.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.966+0000 7facde59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facd8034da0 0x7facd80c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.967+0000 7facddd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd8035180 0x7facd815d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.967+0000 7facded9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd815d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facded9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd815d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:53384/0 (socket says 172.21.15.170:53384) 2023-10-30T17:32:39.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facded9d640 1 -- 172.21.15.170:0/3901805742 learned_addr learned my addr 172.21.15.170:0/3901805742 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facded9d640 10 monclient(hunting): get_auth_request con 0x7facd80bfc20 auth_method 0 2023-10-30T17:32:39.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facded9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facded9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facddd9b640 10 monclient(hunting): get_auth_request con 0x7facd8035180 auth_method 0 2023-10-30T17:32:39.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facddd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facddd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.968+0000 7facded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.969+0000 7facded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7facddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7facddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7facddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7efd6dc74640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd68145a20 0x7efd680afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7facde59c640 10 monclient(hunting): get_auth_request con 0x7facd8034da0 auth_method 0 2023-10-30T17:32:39.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7facde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7facde59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7facde59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7f57427fc640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57440aaed0 0x7f574413f9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f57340098c0 tx=0x7f57340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7f572b7fe640 10 client.?.objecter ms_handle_connect 0x7f57440aaed0 2023-10-30T17:32:39.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.970+0000 7f572b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:39.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facded9d640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2023-10-30T17:32:39.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facded9d640 1 -- 172.21.15.170:0/3901805742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd8035180 msgr2=0x7facd815d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facded9d640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd8035180 0x7facd815d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facded9d640 1 -- 172.21.15.170:0/3901805742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facd8034da0 msgr2=0x7facd80c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facded9d640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facd8034da0 0x7facd80c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facded9d640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facded9d640 1 monclient: found mon.noname-c 2023-10-30T17:32:39.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facded9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facded9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.971+0000 7facded9d640 1 -- 172.21.15.170:0/3901805742 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd0009400 con 0x7facd80bfc20 2023-10-30T17:32:39.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.973+0000 7efd6dc74640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd68145a20 0x7efd680afbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60650/0 (socket says 172.21.15.170:60650) 2023-10-30T17:32:39.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.973+0000 7efd6dc74640 1 -- 172.21.15.170:0/4126556675 learned_addr learned my addr 172.21.15.170:0/4126556675 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.973+0000 7efd6dc74640 10 monclient(hunting): get_auth_request con 0x7efd68145a20 auth_method 0 2023-10-30T17:32:39.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.973+0000 7efd6dc74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.973+0000 7efd6dc74640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.973+0000 7efd6dc74640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.973+0000 7facded9d640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd815d640 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7face0061be0 tx=0x7face0000b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.973+0000 7facceffd640 10 client.?.objecter ms_handle_connect 0x7facd80bfc20 2023-10-30T17:32:39.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7facceffd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:39.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 1 -- 172.21.15.170:0/896178659 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c0317e0 con 0x7f608013eb60 2023-10-30T17:32:39.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:39.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 10 monclient: dump: 2023-10-30T17:32:39.170 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.171 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.171 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.171 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.171 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.171 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.172 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.172 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.172 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.172 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 1 -- 172.21.15.170:0/896178659 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606c031940 con 0x7f608013eb60 2023-10-30T17:32:39.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f6086da5640 5 monclient: authenticate success, global_id 4330 2023-10-30T17:32:39.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f6086da5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:39.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f6086da5640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:39.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f6086da5640 4 set_mon_vals no callback set 2023-10-30T17:32:39.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 1 -- 172.21.15.170:0/896178659 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c031cd0 con 0x7f608013eb60 2023-10-30T17:32:39.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:39.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.974+0000 7f607f7fe640 10 monclient: dump: 2023-10-30T17:32:39.175 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.175 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.175 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.175 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.175 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.176 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.176 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.176 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.176 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.176 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 10 monclient: shutdown 2023-10-30T17:32:39.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 1 -- 172.21.15.170:0/896178659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f608013eb60 msgr2=0x7f608013ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 1 --2- 172.21.15.170:0/896178659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f608013eb60 0x7f608013ef80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f60800b82b0 tx=0x7f606c02b880 comp rx=0 tx=0).stop 2023-10-30T17:32:39.178 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7efd6dc74640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.178 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7efd6dc74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.178 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7efd6dc74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.179 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 1 -- 172.21.15.170:0/896178659 shutdown_connections 2023-10-30T17:32:39.179 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 1 --2- 172.21.15.170:0/896178659 >> v2:172.21.15.170:3300/0 conn(0x7f6080144f50 0x7f60801453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 1 --2- 172.21.15.170:0/896178659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f608013eb60 0x7f608013ef80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 1 --2- 172.21.15.170:0/896178659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60800b7c90 0x7f60800a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 1 -- 172.21.15.170:0/896178659 >> 172.21.15.170:0/896178659 conn(0x7f6080071da0 msgr2=0x7f60800b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 1 -- 172.21.15.170:0/896178659 shutdown_connections 2023-10-30T17:32:39.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.975+0000 7f6086da5640 1 -- 172.21.15.170:0/896178659 wait complete. 2023-10-30T17:32:39.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.976+0000 7f6086da5640 10 monclient: build_initial_monmap 2023-10-30T17:32:39.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.976+0000 7f6086da5640 10 monclient: monmap: 2023-10-30T17:32:39.181 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:39.181 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:39.181 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:39.182 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:39.182 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:39.182 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.182 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:39.182 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:39.182 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:39.183 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.976+0000 7f6086da5640 1 librados: starting msgr at 2023-10-30T17:32:39.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.976+0000 7f6086da5640 1 librados: starting objecter 2023-10-30T17:32:39.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.976+0000 7f6086da5640 1 Processor -- start 2023-10-30T17:32:39.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.976+0000 7f6086da5640 1 -- start start 2023-10-30T17:32:39.184 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.976+0000 7f6086da5640 1 librados: setting wanted keys 2023-10-30T17:32:39.184 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.976+0000 7f6086da5640 1 librados: calling monclient init 2023-10-30T17:32:39.184 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.976+0000 7f6086da5640 10 monclient: init 2023-10-30T17:32:39.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 1 -- 172.21.15.170:0/2086724736 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8003dd0 con 0x7ff9e00b36a0 2023-10-30T17:32:39.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:39.185 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 10 monclient: dump: 2023-10-30T17:32:39.186 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.186 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.186 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.186 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.186 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.187 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.187 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.187 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.187 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.187 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 1 -- 172.21.15.170:0/2086724736 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d800ce30 con 0x7ff9e00b36a0 2023-10-30T17:32:39.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 1 -- 172.21.15.170:0/2086724736 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d80052b0 con 0x7ff9e00b36a0 2023-10-30T17:32:39.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:39.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9c7fff640 10 monclient: dump: 2023-10-30T17:32:39.189 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.189 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.190 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.190 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.190 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.190 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.190 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.190 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.191 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.191 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9e4a21640 5 monclient: authenticate success, global_id 4336 2023-10-30T17:32:39.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9e4a21640 10 monclient: _renew_subs 2023-10-30T17:32:39.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9e4a21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9e4a21640 1 -- 172.21.15.170:0/2086724736 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e00ad5c0 con 0x7ff9e00b36a0 2023-10-30T17:32:39.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9ddffb640 4 set_mon_vals no callback set 2023-10-30T17:32:39.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9e4a21640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:39.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9e4a21640 10 monclient: _renew_subs 2023-10-30T17:32:39.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9e4a21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9e4a21640 1 -- 172.21.15.170:0/2086724736 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e0138620 con 0x7ff9e00b36a0 2023-10-30T17:32:39.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7ff9e4a21640 1 librados: init done 2023-10-30T17:32:39.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7f6086da5640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:39.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7f6086da5640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T17:32:39.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7f6086da5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60800b7c90 0x7f60801428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7f6086da5640 10 monclient(hunting): picked mon.noname-b con 0x7f60800b7c90 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:39.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7f6086da5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f608013eb60 0x7f6080143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7f6086da5640 10 monclient(hunting): picked mon.noname-c con 0x7f608013eb60 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:39.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7f6086da5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6080144f50 0x7f60801d6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7f6086da5640 10 monclient(hunting): picked mon.noname-a con 0x7f6080144f50 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:39.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.977+0000 7f6086da5640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7f6086da5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60801d68a0 con 0x7f6080144f50 2023-10-30T17:32:39.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7f6086da5640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.197 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7f6086da5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608013fb90 con 0x7f60800b7c90 2023-10-30T17:32:39.197 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7f6086da5640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.197 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7f6086da5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6080143890 con 0x7f608013eb60 2023-10-30T17:32:39.197 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7f6086da5640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:39.197 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7f6086da5640 10 monclient(hunting): authenticate will time out at 1341.301173s 2023-10-30T17:32:39.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7efd6cc72640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd680a8b00 0x7efd681db9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7efd6cc72640 10 monclient(hunting): get_auth_request con 0x7efd680a8b00 auth_method 0 2023-10-30T17:32:39.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7efd6cc72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7efd6cc72640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7efd6cc72640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7ff9e4a21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7ff9e4a21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7ff9e4a21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.978+0000 7ff9e4a21640 1 -- 172.21.15.170:0/2086724736 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e00ad7b0 con 0x7ff9e00b36a0 2023-10-30T17:32:39.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.979+0000 7efd6cc72640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.979+0000 7efd6cc72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.979+0000 7efd6cc72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7efd6dc74640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2023-10-30T17:32:39.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7efd6dc74640 1 -- 172.21.15.170:0/4126556675 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd680a8b00 msgr2=0x7efd681db9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7efd6dc74640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd680a8b00 0x7efd681db9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7efd6dc74640 1 -- 172.21.15.170:0/4126556675 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd680a8720 msgr2=0x7efd68138520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7efd6dc74640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd680a8720 0x7efd68138520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7efd6dc74640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7efd6dc74640 1 monclient: found mon.noname-b 2023-10-30T17:32:39.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7efd6dc74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.203 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7efd6dc74640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.203 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7efd6dc74640 1 -- 172.21.15.170:0/4126556675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd58009400 con 0x7efd68145a20 2023-10-30T17:32:39.203 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.980+0000 7f778259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77840b7c90 0x7f7784144540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.204 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f778259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77840b7c90 0x7f7784144540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60652/0 (socket says 172.21.15.170:60652) 2023-10-30T17:32:39.204 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f778259c640 1 -- 172.21.15.170:0/353202548 learned_addr learned my addr 172.21.15.170:0/353202548 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.204 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f778259c640 10 monclient(hunting): get_auth_request con 0x7f77840b7c90 auth_method 0 2023-10-30T17:32:39.204 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f778259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.204 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f778259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f778259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f7782d9d640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 0x7f77841d65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 1 -- 172.21.15.170:0/3872762520 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec02ca70 con 0x7f58f40a8b00 2023-10-30T17:32:39.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:39.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 10 monclient: dump: 2023-10-30T17:32:39.206 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.207 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.207 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.207 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.207 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.207 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.207 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.208 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.208 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.208 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 1 -- 172.21.15.170:0/3872762520 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ec02cbd0 con 0x7f58f40a8b00 2023-10-30T17:32:39.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 1 -- 172.21.15.170:0/3872762520 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec0343f0 con 0x7f58f40a8b00 2023-10-30T17:32:39.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:39.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58f88db640 10 monclient: dump: 2023-10-30T17:32:39.210 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.210 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.210 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.210 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.211 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.211 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.211 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.211 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.211 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.212 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58fbb41640 5 monclient: authenticate success, global_id 4326 2023-10-30T17:32:39.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58fbb41640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:39.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.982+0000 7f58fbb41640 4 set_mon_vals no callback set 2023-10-30T17:32:39.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7efd6dc74640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd68145a20 0x7efd680afbf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efd6000b970 tx=0x7efd6000be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7efd5dffb640 10 client.?.objecter ms_handle_connect 0x7efd68145a20 2023-10-30T17:32:39.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7efd5dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:39.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 10 monclient: shutdown 2023-10-30T17:32:39.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 1 -- 172.21.15.170:0/3872762520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58f40a8b00 msgr2=0x7f58f4145a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 1 --2- 172.21.15.170:0/3872762520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58f40a8b00 0x7f58f4145a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f58f4067150 tx=0x7f58ec02bed0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 1 -- 172.21.15.170:0/3872762520 shutdown_connections 2023-10-30T17:32:39.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 1 --2- 172.21.15.170:0/3872762520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58f40a8b00 0x7f58f4145a20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 1 --2- 172.21.15.170:0/3872762520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f413f4d0 0x7f58f41454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 1 --2- 172.21.15.170:0/3872762520 >> v2:172.21.15.177:3300/0 conn(0x7f58f40a8720 0x7f58f413ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 1 -- 172.21.15.170:0/3872762520 >> 172.21.15.170:0/3872762520 conn(0x7f58f40b7300 msgr2=0x7f58f40b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 1 -- 172.21.15.170:0/3872762520 shutdown_connections 2023-10-30T17:32:39.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f58fbb41640 1 -- 172.21.15.170:0/3872762520 wait complete. 2023-10-30T17:32:39.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 1 -- 172.21.15.170:0/3951768484 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c031870 con 0x7fc88c0b7c90 2023-10-30T17:32:39.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:39.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 10 monclient: dump: 2023-10-30T17:32:39.217 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.217 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.217 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.218 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.218 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.218 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.218 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.218 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.219 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.219 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 1 -- 172.21.15.170:0/3951768484 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c0319d0 con 0x7fc88c0b7c90 2023-10-30T17:32:39.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc8925c9640 5 monclient: authenticate success, global_id 4329 2023-10-30T17:32:39.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc8925c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:39.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 1 -- 172.21.15.170:0/3951768484 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c031d60 con 0x7fc88c0b7c90 2023-10-30T17:32:39.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:39.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7fc88affd640 10 monclient: dump: 2023-10-30T17:32:39.221 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.221 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.221 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.222 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.222 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.222 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.222 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.222 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.223 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.223 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7fc8925c9640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:39.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7fc8925c9640 4 set_mon_vals no callback set 2023-10-30T17:32:39.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f778259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f778259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.983+0000 7f778259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7f58fbb41640 10 monclient: build_initial_monmap 2023-10-30T17:32:39.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7f58fbb41640 10 monclient: monmap: 2023-10-30T17:32:39.225 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:39.225 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:39.225 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:39.225 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:39.226 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:39.226 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.226 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:39.226 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:39.226 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:39.227 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7f58fbb41640 1 librados: starting msgr at 2023-10-30T17:32:39.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7f58fbb41640 1 librados: starting objecter 2023-10-30T17:32:39.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7f58fbb41640 1 Processor -- start 2023-10-30T17:32:39.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7f58fbb41640 1 -- start start 2023-10-30T17:32:39.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7f58fbb41640 1 librados: setting wanted keys 2023-10-30T17:32:39.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7f58fbb41640 1 librados: calling monclient init 2023-10-30T17:32:39.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7f58fbb41640 10 monclient: init 2023-10-30T17:32:39.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7fc8925c9640 10 monclient: shutdown 2023-10-30T17:32:39.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7fc8925c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7fc8925c9640 1 -- 172.21.15.170:0/3951768484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c0b7c90 msgr2=0x7fc88c0a8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7fc8925c9640 1 --2- 172.21.15.170:0/3951768484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c0b7c90 0x7fc88c0a8b80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0aae00 tx=0x7fc87c02b910 comp rx=0 tx=0).stop 2023-10-30T17:32:39.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7fc8925c9640 1 -- 172.21.15.170:0/3951768484 shutdown_connections 2023-10-30T17:32:39.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7fc8925c9640 1 --2- 172.21.15.170:0/3951768484 >> v2:172.21.15.177:3300/0 conn(0x7fc88c144f50 0x7fc88c145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7fc8925c9640 1 --2- 172.21.15.170:0/3951768484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c0b7c90 0x7fc88c0a8b80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7fc8925c9640 1 -- 172.21.15.170:0/3951768484 >> 172.21.15.170:0/3951768484 conn(0x7fc88c071da0 msgr2=0x7fc88c0b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.984+0000 7f7781d9b640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f778413eb60 0x7f77841438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7fc8925c9640 1 -- 172.21.15.170:0/3951768484 shutdown_connections 2023-10-30T17:32:39.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7fc8925c9640 1 -- 172.21.15.170:0/3951768484 wait complete. 2023-10-30T17:32:39.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:39.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T17:32:39.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f40a8720 0x7f58f40b3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 10 monclient(hunting): picked mon.noname-b con 0x7f58f40a8720 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:39.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f40a8b00 0x7f58f40b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 10 monclient(hunting): picked mon.noname-c con 0x7f58f40a8b00 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:39.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58f413f4d0 0x7f58f40ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.234 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 10 monclient(hunting): picked mon.noname-a con 0x7f58f413f4d0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:39.234 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.234 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40abcb0 con 0x7f58f413f4d0 2023-10-30T17:32:39.234 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.234 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40ace20 con 0x7f58f40a8720 2023-10-30T17:32:39.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40b49d0 con 0x7f58f40a8b00 2023-10-30T17:32:39.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:39.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 1 -- 172.21.15.170:0/3901805742 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0072240 con 0x7facd80bfc20 2023-10-30T17:32:39.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:39.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 10 monclient: dump: 2023-10-30T17:32:39.236 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.236 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.237 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.237 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.237 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.237 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.237 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.238 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.238 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.238 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7f58fbb41640 10 monclient(hunting): authenticate will time out at 1341.308619s 2023-10-30T17:32:39.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 1 -- 172.21.15.170:0/3901805742 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face00723a0 con 0x7facd80bfc20 2023-10-30T17:32:39.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 1 -- 172.21.15.170:0/3901805742 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face00923b0 con 0x7facd80bfc20 2023-10-30T17:32:39.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:39.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facceffd640 10 monclient: dump: 2023-10-30T17:32:39.241 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.241 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.241 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.241 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.242 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.242 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.242 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.242 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.242 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.243 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.243 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7facdcd99640 4 set_mon_vals no callback set 2023-10-30T17:32:39.243 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7face4cf2640 5 monclient: authenticate success, global_id 4298 2023-10-30T17:32:39.243 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7face4cf2640 10 monclient: _renew_subs 2023-10-30T17:32:39.243 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7face4cf2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7face4cf2640 1 -- 172.21.15.170:0/3901805742 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd815db80 con 0x7facd80bfc20 2023-10-30T17:32:39.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7face4cf2640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:39.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7face4cf2640 10 monclient: _renew_subs 2023-10-30T17:32:39.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7face4cf2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7face4cf2640 1 -- 172.21.15.170:0/3901805742 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd80c49f0 con 0x7facd80bfc20 2023-10-30T17:32:39.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.985+0000 7face4cf2640 1 librados: init done 2023-10-30T17:32:39.245 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7fc8925c9640 10 monclient: build_initial_monmap 2023-10-30T17:32:39.245 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7fc8925c9640 10 monclient: monmap: 2023-10-30T17:32:39.245 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:39.246 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:39.246 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:39.246 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:39.246 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:39.246 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.247 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:39.247 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:39.247 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:39.247 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7f607ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f608013eb60 0x7f6080143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7fc8925c9640 1 librados: starting msgr at 2023-10-30T17:32:39.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7fc8925c9640 1 librados: starting objecter 2023-10-30T17:32:39.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7fc8925c9640 1 Processor -- start 2023-10-30T17:32:39.249 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7face4cf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.249 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7face4cf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.249 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7face4cf2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7face4cf2640 1 -- 172.21.15.170:0/3901805742 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd815ce20 con 0x7facd80bfc20 2023-10-30T17:32:39.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7fc8925c9640 1 -- start start 2023-10-30T17:32:39.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7fc8925c9640 1 librados: setting wanted keys 2023-10-30T17:32:39.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7fc8925c9640 1 librados: calling monclient init 2023-10-30T17:32:39.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7fc8925c9640 10 monclient: init 2023-10-30T17:32:39.251 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7f7782d9d640 10 monclient(hunting): get_auth_request con 0x7f7784144f50 auth_method 0 2023-10-30T17:32:39.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7f7782d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7f7782d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.986+0000 7f7782d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f7781d9b640 10 monclient(hunting): get_auth_request con 0x7f778413eb60 auth_method 0 2023-10-30T17:32:39.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f7781d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.252 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f7781d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f7781d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f607ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f608013eb60 0x7f6080143350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:53410/0 (socket says 172.21.15.170:53410) 2023-10-30T17:32:39.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f607ffff640 1 -- 172.21.15.170:0/1716984697 learned_addr learned my addr 172.21.15.170:0/1716984697 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f778259c640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2023-10-30T17:32:39.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f778259c640 1 -- 172.21.15.170:0/353202548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 msgr2=0x7f77841d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f778259c640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 0x7f77841d65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f778259c640 1 -- 172.21.15.170:0/353202548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f778413eb60 msgr2=0x7f77841438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f778259c640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f778413eb60 0x7f77841438a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f778259c640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f778259c640 1 monclient: found mon.noname-b 2023-10-30T17:32:39.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f778259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f778259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f778259c640 1 -- 172.21.15.170:0/353202548 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c009a70 con 0x7f77840b7c90 2023-10-30T17:32:39.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f607ffff640 10 monclient(hunting): get_auth_request con 0x7f608013eb60 auth_method 0 2023-10-30T17:32:39.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:39.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:32:39.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c0b7c90 0x7fc88c13a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f607ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f607ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc88c0b7c90 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:39.259 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc88c13eb60 0x7fc88c13ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.259 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 10 monclient(hunting): picked mon.noname-b con 0x7fc88c13eb60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:39.259 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c144f50 0x7fc88c135470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.259 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7f607ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 10 monclient(hunting): picked mon.noname-a con 0x7fc88c144f50 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:39.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c13be50 con 0x7fc88c144f50 2023-10-30T17:32:39.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c139d40 con 0x7fc88c13eb60 2023-10-30T17:32:39.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:39.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c1399d0 con 0x7fc88c0b7c90 2023-10-30T17:32:39.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:39.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.987+0000 7fc8925c9640 10 monclient(hunting): authenticate will time out at 1341.310711s 2023-10-30T17:32:39.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 1 -- 172.21.15.170:0/3889062015 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573403a6f0 con 0x7f57440aaed0 2023-10-30T17:32:39.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:39.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 10 monclient: dump: 2023-10-30T17:32:39.264 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.264 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.264 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.264 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.264 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.264 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.265 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.265 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.265 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.265 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.265 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 1 -- 172.21.15.170:0/3889062015 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573403a870 con 0x7f57440aaed0 2023-10-30T17:32:39.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f5749483640 5 monclient: authenticate success, global_id 4332 2023-10-30T17:32:39.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 1 -- 172.21.15.170:0/3889062015 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573403ac20 con 0x7f57440aaed0 2023-10-30T17:32:39.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:39.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f57417fa640 4 set_mon_vals no callback set 2023-10-30T17:32:39.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f572b7fe640 10 monclient: dump: 2023-10-30T17:32:39.267 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.268 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.268 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.268 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.268 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.268 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.269 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.269 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.269 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.269 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.270 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f607ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.270 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f607ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.270 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f607ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f5749483640 10 monclient: _renew_subs 2023-10-30T17:32:39.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f5749483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f5749483640 1 -- 172.21.15.170:0/3889062015 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5744140040 con 0x7f57440aaed0 2023-10-30T17:32:39.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f5749483640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:39.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f5749483640 10 monclient: _renew_subs 2023-10-30T17:32:39.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f5749483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f5749483640 1 -- 172.21.15.170:0/3889062015 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5744140ab0 con 0x7f57440aaed0 2023-10-30T17:32:39.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.989+0000 7f5749483640 1 librados: init done 2023-10-30T17:32:39.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.990+0000 7f778259c640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77840b7c90 0x7f7784144540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f777400c500 tx=0x7f777400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.990+0000 7f7772ffd640 10 client.?.objecter ms_handle_connect 0x7f77840b7c90 2023-10-30T17:32:39.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.990+0000 7f7772ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:39.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.990+0000 7f5749483640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.990+0000 7f5749483640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.990+0000 7f5749483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.990+0000 7f5749483640 1 -- 172.21.15.170:0/3889062015 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5744140200 con 0x7f57440aaed0 2023-10-30T17:32:39.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.990+0000 7f6085342640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6080144f50 0x7f60801d6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.991+0000 7f58f90dc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f40a8b00 0x7f58f40b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.991+0000 7f58f90dc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f40a8b00 0x7f58f40b5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:53416/0 (socket says 172.21.15.170:53416) 2023-10-30T17:32:39.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.991+0000 7f58f90dc640 1 -- 172.21.15.170:0/3308378740 learned_addr learned my addr 172.21.15.170:0/3308378740 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.991+0000 7f58f90dc640 10 monclient(hunting): get_auth_request con 0x7f58f40a8b00 auth_method 0 2023-10-30T17:32:39.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.991+0000 7f58f90dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.991+0000 7f58f90dc640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.991+0000 7f58f90dc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.992+0000 7f6085342640 10 monclient(hunting): get_auth_request con 0x7f6080144f50 auth_method 0 2023-10-30T17:32:39.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.992+0000 7f6085342640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.992+0000 7f6085342640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.992+0000 7f6085342640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.992+0000 7f58f90dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.992+0000 7f58f90dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.992+0000 7f58f90dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.993+0000 7f58fa0de640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58f413f4d0 0x7f58f40ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.993+0000 7f6085342640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.993+0000 7f6085342640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.993+0000 7f6085342640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.996+0000 7f6085342640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2023-10-30T17:32:39.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.996+0000 7f6085342640 1 -- 172.21.15.170:0/1716984697 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f608013eb60 msgr2=0x7f6080143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.996+0000 7f6085342640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f608013eb60 0x7f6080143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.996+0000 7f6085342640 1 -- 172.21.15.170:0/1716984697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60800b7c90 msgr2=0x7f60801428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.996+0000 7f6085342640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60800b7c90 0x7f60801428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.996+0000 7f6085342640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.996+0000 7f6085342640 1 monclient: found mon.noname-a 2023-10-30T17:32:39.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.996+0000 7f6085342640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.996+0000 7f6085342640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.996+0000 7f6085342640 1 -- 172.21.15.170:0/1716984697 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c0094d0 con 0x7f6080144f50 2023-10-30T17:32:39.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.998+0000 7f58f90dc640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2023-10-30T17:32:39.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.998+0000 7f58f90dc640 1 -- 172.21.15.170:0/3308378740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f40a8720 msgr2=0x7f58f40b3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.998+0000 7f58f90dc640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f40a8720 0x7f58f40b3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.998+0000 7f58f90dc640 1 -- 172.21.15.170:0/3308378740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58f413f4d0 msgr2=0x7f58f40ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.998+0000 7f58f90dc640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58f413f4d0 0x7f58f40ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.998+0000 7f58f90dc640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.998+0000 7f58f90dc640 1 monclient: found mon.noname-c 2023-10-30T17:32:39.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.998+0000 7f58f90dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.998+0000 7f58f90dc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:38.998+0000 7f58f90dc640 1 -- 172.21.15.170:0/3308378740 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec009b30 con 0x7f58f40a8b00 2023-10-30T17:32:39.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.058+0000 7f572b7fe640 1 -- 172.21.15.170:0/3889062015 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f573404c3b0 con 0x7f57440aaed0 2023-10-30T17:32:39.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.059+0000 7f572b7fe640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f573006c880 0x7f573006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.059+0000 7f5742ffd640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f573006c880 0x7f573006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.059+0000 7f5742ffd640 10 monclient: get_auth_request con 0x7f573006c880 auth_method 0 2023-10-30T17:32:39.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.060+0000 7f5742ffd640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f573006c880 0x7f573006ed40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f573c009ce0 tx=0x7f573c00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.060+0000 7f572b7fe640 10 client.4332.objecter ms_handle_connect 0x7f573006c880 2023-10-30T17:32:39.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 1 -- 172.21.15.170:0/4126556675 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6000cb40 con 0x7efd68145a20 2023-10-30T17:32:39.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:39.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 10 monclient: dump: 2023-10-30T17:32:39.288 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.289 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.289 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.289 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.289 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.289 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.290 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.290 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.290 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.290 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 1 -- 172.21.15.170:0/4126556675 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6000ccc0 con 0x7efd68145a20 2023-10-30T17:32:39.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd6f6d7640 5 monclient: authenticate success, global_id 4345 2023-10-30T17:32:39.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd6f6d7640 10 monclient: _renew_subs 2023-10-30T17:32:39.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd6f6d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd6f6d7640 1 -- 172.21.15.170:0/4126556675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd68138a60 con 0x7efd68145a20 2023-10-30T17:32:39.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 1 -- 172.21.15.170:0/4126556675 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd60014680 con 0x7efd68145a20 2023-10-30T17:32:39.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:39.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5dffb640 10 monclient: dump: 2023-10-30T17:32:39.293 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.293 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.293 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.293 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.293 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.294 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.294 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.294 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.294 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.294 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd5f7fe640 4 set_mon_vals no callback set 2023-10-30T17:32:39.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd6f6d7640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:39.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd6f6d7640 10 monclient: _renew_subs 2023-10-30T17:32:39.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd6f6d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd6f6d7640 1 -- 172.21.15.170:0/4126556675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd680abd60 con 0x7efd68145a20 2023-10-30T17:32:39.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.227+0000 7efd6f6d7640 1 librados: init done 2023-10-30T17:32:39.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.229+0000 7efd6f6d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.229+0000 7efd6f6d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.229+0000 7efd6f6d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.229+0000 7efd6f6d7640 1 -- 172.21.15.170:0/4126556675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd681db2e0 con 0x7efd68145a20 2023-10-30T17:32:39.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 1 -- 172.21.15.170:0/353202548 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774005ce0 con 0x7f77840b7c90 2023-10-30T17:32:39.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:39.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 10 monclient: dump: 2023-10-30T17:32:39.298 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.298 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.299 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.299 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.299 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.299 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.299 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.299 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.300 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.300 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 1 -- 172.21.15.170:0/353202548 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774005e40 con 0x7f77840b7c90 2023-10-30T17:32:39.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 1 -- 172.21.15.170:0/353202548 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774007330 con 0x7f77840b7c90 2023-10-30T17:32:39.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:39.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7772ffd640 10 monclient: dump: 2023-10-30T17:32:39.302 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.302 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.302 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.302 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.302 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.303 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.303 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.303 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.303 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.303 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7788851640 5 monclient: authenticate success, global_id 4348 2023-10-30T17:32:39.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7788851640 10 monclient: _renew_subs 2023-10-30T17:32:39.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7780d99640 4 set_mon_vals no callback set 2023-10-30T17:32:39.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7788851640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7788851640 1 -- 172.21.15.170:0/353202548 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77840a99c0 con 0x7f77840b7c90 2023-10-30T17:32:39.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7788851640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:39.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7788851640 10 monclient: _renew_subs 2023-10-30T17:32:39.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7788851640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7788851640 1 -- 172.21.15.170:0/353202548 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77841d6af0 con 0x7f77840b7c90 2023-10-30T17:32:39.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.246+0000 7f7788851640 1 librados: init done 2023-10-30T17:32:39.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.247+0000 7f7788851640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.247+0000 7f7788851640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.247+0000 7f7788851640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.247+0000 7f7788851640 1 -- 172.21.15.170:0/353202548 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77841d62d0 con 0x7f77840b7c90 2023-10-30T17:32:39.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.254+0000 7facceffd640 1 -- 172.21.15.170:0/3901805742 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7face0092510 con 0x7facd80bfc20 2023-10-30T17:32:39.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.255+0000 7facceffd640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7facc006c850 0x7facc006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.255+0000 7facde59c640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7facc006c850 0x7facc006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.255+0000 7facde59c640 10 monclient: get_auth_request con 0x7facc006c850 auth_method 0 2023-10-30T17:32:39.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.255+0000 7facceffd640 10 client.4298.objecter ms_handle_connect 0x7facc006c850 2023-10-30T17:32:39.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.255+0000 7facde59c640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7facc006c850 0x7facc006ed10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facd40017f0 tx=0x7facd4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.259+0000 7ff9c7fff640 1 -- 172.21.15.170:0/2086724736 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff9d8005410 con 0x7ff9e00b36a0 2023-10-30T17:32:39.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.260+0000 7ff9c7fff640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff9cc06c880 0x7ff9cc06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.260+0000 7ff9df7fe640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff9cc06c880 0x7ff9cc06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.260+0000 7ff9df7fe640 10 monclient: get_auth_request con 0x7ff9cc06c880 auth_method 0 2023-10-30T17:32:39.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.260+0000 7ff9df7fe640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff9cc06c880 0x7ff9cc06ed40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009670 tx=0x7ff9d4006bf0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.260+0000 7ff9c7fff640 10 client.4336.objecter ms_handle_connect 0x7ff9cc06c880 2023-10-30T17:32:39.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.261+0000 7ff9c7fff640 1 -- 172.21.15.170:0/2086724736 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff9d8057fe0 con 0x7ff9e00b36a0 2023-10-30T17:32:39.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.261+0000 7ff9c7fff640 10 client.4336.objecter ms_dispatch 0x7ff9e0000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:39.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.261+0000 7ff9c7fff640 3 client.4336.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:39.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.261+0000 7ff9c7fff640 3 client.4336.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:39.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.261+0000 7ff9c7fff640 20 client.4336.objecter dump_active .. 0 homeless 2023-10-30T17:32:39.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.262+0000 7f572b7fe640 1 -- 172.21.15.170:0/3889062015 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f573407ad70 con 0x7f57440aaed0 2023-10-30T17:32:39.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.262+0000 7f572b7fe640 10 client.4332.objecter ms_dispatch 0x7f5744000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:39.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.262+0000 7f572b7fe640 3 client.4332.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:39.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.262+0000 7f572b7fe640 3 client.4332.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:39.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.262+0000 7f572b7fe640 20 client.4332.objecter dump_active .. 0 homeless 2023-10-30T17:32:39.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.262+0000 7f572b7fe640 1 -- 172.21.15.170:0/3889062015 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f57340769b0 con 0x7f57440aaed0 2023-10-30T17:32:39.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.262+0000 7f572b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.262+0000 7f572b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:39.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.265+0000 7f6085342640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6080144f50 0x7f60801d6d30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f607800be20 tx=0x7f607800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.265+0000 7f607d7fa640 10 client.?.objecter ms_handle_connect 0x7f6080144f50 2023-10-30T17:32:39.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.265+0000 7f607d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:39.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.267+0000 7facceffd640 1 -- 172.21.15.170:0/3901805742 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7face00d1d80 con 0x7facd80bfc20 2023-10-30T17:32:39.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.267+0000 7facceffd640 10 client.4298.objecter ms_dispatch 0x7face0000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:39.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.267+0000 7facceffd640 3 client.4298.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:39.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.267+0000 7facceffd640 3 client.4298.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:39.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.268+0000 7facceffd640 20 client.4298.objecter dump_active .. 0 homeless 2023-10-30T17:32:39.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.268+0000 7facceffd640 1 -- 172.21.15.170:0/3901805742 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7face00ce0c0 con 0x7facd80bfc20 2023-10-30T17:32:39.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.268+0000 7facceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.268+0000 7facceffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:39.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.269+0000 7f58f90dc640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f40a8b00 0x7f58f40b5a40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58e400c7f0 tx=0x7f58e400ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.269+0000 7f58ea7fc640 10 client.?.objecter ms_handle_connect 0x7f58f40a8b00 2023-10-30T17:32:39.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.269+0000 7f58ea7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:39.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.271+0000 7fc88b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc88c13eb60 0x7fc88c13ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.271+0000 7ff9c7fff640 1 -- 172.21.15.170:0/2086724736 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9d8054320 con 0x7ff9e00b36a0 2023-10-30T17:32:39.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.271+0000 7ff9c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.271+0000 7ff9c7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:39.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.271+0000 7fc88bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c0b7c90 0x7fc88c13a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.272+0000 7fc88b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc88c13eb60 0x7fc88c13ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:60692/0 (socket says 172.21.15.170:60692) 2023-10-30T17:32:39.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.272+0000 7fc88b7fe640 1 -- 172.21.15.170:0/2799804909 learned_addr learned my addr 172.21.15.170:0/2799804909 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:39.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.272+0000 7fc88b7fe640 10 monclient(hunting): get_auth_request con 0x7fc88c13eb60 auth_method 0 2023-10-30T17:32:39.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.272+0000 7fc88b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.272+0000 7fc88b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.272+0000 7fc88b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.273+0000 7fc88bfff640 10 monclient(hunting): get_auth_request con 0x7fc88c0b7c90 auth_method 0 2023-10-30T17:32:39.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.273+0000 7fc88bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:39.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.273+0000 7fc88bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:39.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.273+0000 7fc88bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:39.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.274+0000 7fc88bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:39.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.274+0000 7fc88bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:39.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.274+0000 7fc88bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:39.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.277+0000 7fc88bfff640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2023-10-30T17:32:39.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.277+0000 7fc88bfff640 1 -- 172.21.15.170:0/2799804909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc88c13eb60 msgr2=0x7fc88c13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.277+0000 7fc88bfff640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc88c13eb60 0x7fc88c13ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.277+0000 7fc88bfff640 1 -- 172.21.15.170:0/2799804909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c144f50 msgr2=0x7fc88c135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.277+0000 7fc88bfff640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c144f50 0x7fc88c135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.277+0000 7fc88bfff640 10 monclient: _finish_hunting 0 2023-10-30T17:32:39.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.277+0000 7fc88bfff640 1 monclient: found mon.noname-c 2023-10-30T17:32:39.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.277+0000 7fc88bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:39.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.277+0000 7fc88bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.277+0000 7fc88bfff640 1 -- 172.21.15.170:0/2799804909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c0094d0 con 0x7fc88c0b7c90 2023-10-30T17:32:39.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.280+0000 7fc88bfff640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c0b7c90 0x7fc88c13a6e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc87c033000 tx=0x7fc87c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.280+0000 7fc888ff9640 10 client.?.objecter ms_handle_connect 0x7fc88c0b7c90 2023-10-30T17:32:39.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.280+0000 7fc888ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:39.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 1 -- 172.21.15.170:0/1716984697 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607800ce70 con 0x7f6080144f50 2023-10-30T17:32:39.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:39.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 10 monclient: dump: 2023-10-30T17:32:39.326 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.326 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.326 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.328 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 1 -- 172.21.15.170:0/1716984697 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6078014440 con 0x7f6080144f50 2023-10-30T17:32:39.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 1 -- 172.21.15.170:0/1716984697 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6078014810 con 0x7f6080144f50 2023-10-30T17:32:39.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:39.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607d7fa640 10 monclient: dump: 2023-10-30T17:32:39.330 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.330 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.330 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.330 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.330 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.330 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.331 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.331 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.331 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.331 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f6086da5640 5 monclient: authenticate success, global_id 4341 2023-10-30T17:32:39.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f6086da5640 10 monclient: _renew_subs 2023-10-30T17:32:39.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f6086da5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f6086da5640 1 -- 172.21.15.170:0/1716984697 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6080141a00 con 0x7f6080144f50 2023-10-30T17:32:39.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f607effd640 4 set_mon_vals no callback set 2023-10-30T17:32:39.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f6086da5640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:39.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f6086da5640 10 monclient: _renew_subs 2023-10-30T17:32:39.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f6086da5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f6086da5640 1 -- 172.21.15.170:0/1716984697 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60801d7450 con 0x7f6080144f50 2023-10-30T17:32:39.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.288+0000 7f6086da5640 1 librados: init done 2023-10-30T17:32:39.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.289+0000 7f6086da5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.289+0000 7f6086da5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.289+0000 7f6086da5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.289+0000 7f6086da5640 1 -- 172.21.15.170:0/1716984697 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6080067cd0 con 0x7f6080144f50 2023-10-30T17:32:39.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 1 -- 172.21.15.170:0/3308378740 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4007820 con 0x7f58f40a8b00 2023-10-30T17:32:39.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:39.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 10 monclient: dump: 2023-10-30T17:32:39.336 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.336 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.336 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.336 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.337 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.337 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.337 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.337 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.337 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.337 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 1 -- 172.21.15.170:0/3308378740 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e4007980 con 0x7f58f40a8b00 2023-10-30T17:32:39.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58fbb41640 5 monclient: authenticate success, global_id 4310 2023-10-30T17:32:39.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58fbb41640 10 monclient: _renew_subs 2023-10-30T17:32:39.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58fbb41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58fbb41640 1 -- 172.21.15.170:0/3308378740 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f40ac3c0 con 0x7f58f40a8b00 2023-10-30T17:32:39.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 1 -- 172.21.15.170:0/3308378740 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4007cf0 con 0x7f58f40a8b00 2023-10-30T17:32:39.340 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.340 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:39.340 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58fbb41640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:39.340 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ea7fc640 10 monclient: dump: 2023-10-30T17:32:39.340 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.340 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.341 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.341 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.341 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.341 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.341 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.342 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.342 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.342 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58fbb41640 10 monclient: _renew_subs 2023-10-30T17:32:39.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58fbb41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58fbb41640 1 -- 172.21.15.170:0/3308378740 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f40b60b0 con 0x7f58f40a8b00 2023-10-30T17:32:39.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58fbb41640 1 librados: init done 2023-10-30T17:32:39.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.290+0000 7f58ebfff640 4 set_mon_vals no callback set 2023-10-30T17:32:39.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7f58fbb41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7f58fbb41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7f58fbb41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7f58fbb41640 1 -- 172.21.15.170:0/3308378740 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f40b8140 con 0x7f58f40a8b00 2023-10-30T17:32:39.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 1 -- 172.21.15.170:0/2799804909 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c03ad20 con 0x7fc88c0b7c90 2023-10-30T17:32:39.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:39.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 10 monclient: dump: 2023-10-30T17:32:39.346 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.346 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.346 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.346 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.346 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.347 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.347 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.347 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.347 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.347 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 10 monclient: _finish_auth 0 2023-10-30T17:32:39.348 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 10 monclient: _check_auth_tickets 2023-10-30T17:32:39.348 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:39.348 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 1 -- 172.21.15.170:0/2799804909 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c0443f0 con 0x7fc88c0b7c90 2023-10-30T17:32:39.348 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc8925c9640 5 monclient: authenticate success, global_id 4313 2023-10-30T17:32:39.348 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc8925c9640 10 monclient: _renew_subs 2023-10-30T17:32:39.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc8925c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc8925c9640 1 -- 172.21.15.170:0/2799804909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c142770 con 0x7fc88c0b7c90 2023-10-30T17:32:39.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:39.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 1 -- 172.21.15.170:0/2799804909 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c03aea0 con 0x7fc88c0b7c90 2023-10-30T17:32:39.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:39.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:39.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc888ff9640 10 monclient: dump: 2023-10-30T17:32:39.350 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:39.350 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:39.350 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.351 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:39.351 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:39.351 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:39.351 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:39.351 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:39.352 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:39.352 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:39.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.292+0000 7fc88a7fc640 4 set_mon_vals no callback set 2023-10-30T17:32:39.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.293+0000 7fc8925c9640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:39.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.293+0000 7fc8925c9640 10 monclient: _renew_subs 2023-10-30T17:32:39.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.293+0000 7fc8925c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.293+0000 7fc8925c9640 1 -- 172.21.15.170:0/2799804909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88c135b90 con 0x7fc88c0b7c90 2023-10-30T17:32:39.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.293+0000 7fc8925c9640 1 librados: init done 2023-10-30T17:32:39.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.294+0000 7fc8925c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.294+0000 7fc8925c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.294+0000 7fc8925c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.354 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.294+0000 7fc8925c9640 1 -- 172.21.15.170:0/2799804909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88c1d9c40 con 0x7fc88c0b7c90 2023-10-30T17:32:39.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.454+0000 7f5749483640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T17:32:39.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.454+0000 7f5749483640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T17:32:39.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.454+0000 7f5749483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.454+0000 7f5749483640 1 -- 172.21.15.170:0/3889062015 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f57440a8750 con 0x7f57440aaed0 2023-10-30T17:32:39.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.463+0000 7ff9e4a21640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T17:32:39.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.463+0000 7ff9e4a21640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T17:32:39.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.463+0000 7ff9e4a21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.463+0000 7ff9e4a21640 1 -- 172.21.15.170:0/2086724736 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff9e00af470 con 0x7ff9e00b36a0 2023-10-30T17:32:39.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.484+0000 7face4cf2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T17:32:39.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.484+0000 7face4cf2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T17:32:39.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.484+0000 7face4cf2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:39.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.484+0000 7face4cf2640 1 -- 172.21.15.170:0/3901805742 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7facd80c49f0 con 0x7facd80bfc20 2023-10-30T17:32:39.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.600+0000 7f607d7fa640 1 -- 172.21.15.170:0/1716984697 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f607801d3b0 con 0x7f6080144f50 2023-10-30T17:32:39.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.601+0000 7f607d7fa640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f606406c800 0x7f606406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.603 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.601+0000 7f6084b41640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f606406c800 0x7f606406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.603 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.601+0000 7f6084b41640 10 monclient: get_auth_request con 0x7f606406c800 auth_method 0 2023-10-30T17:32:39.603 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.602+0000 7f607d7fa640 10 client.4341.objecter ms_handle_connect 0x7f606406c800 2023-10-30T17:32:39.603 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.602+0000 7f6084b41640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f606406c800 0x7f606406ecc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6074009710 tx=0x7f6074006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f607d7fa640 1 -- 172.21.15.170:0/1716984697 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f60780589e0 con 0x7f6080144f50 2023-10-30T17:32:39.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f607d7fa640 10 client.4341.objecter ms_dispatch 0x7f6080000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:39.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f607d7fa640 3 client.4341.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:39.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f607d7fa640 3 client.4341.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:39.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f607d7fa640 20 client.4341.objecter dump_active .. 0 homeless 2023-10-30T17:32:39.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f607d7fa640 1 -- 172.21.15.170:0/1716984697 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6078054d20 con 0x7f6080144f50 2023-10-30T17:32:39.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f607d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f607d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:39.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f572b7fe640 1 -- 172.21.15.170:0/3889062015 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5734035090 con 0x7f57440aaed0 2023-10-30T17:32:39.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f572b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T17:32:39.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.621+0000 7f572b7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:39.626 INFO:teuthology.orchestra.run.smithi170.stdout:68719476738 2023-10-30T17:32:39.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 10 librados: watch_flush enter 2023-10-30T17:32:39.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 10 librados: watch_flush exit 2023-10-30T17:32:39.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 20 client.4332.objecter shutdown clearing up homeless session... 2023-10-30T17:32:39.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 10 client.4332.objecter successfully canceled tick 2023-10-30T17:32:39.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 -- 172.21.15.170:0/3889062015 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f573006c880 msgr2=0x7f573006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f573006c880 0x7f573006ed40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f573c009ce0 tx=0x7f573c00b040 comp rx=0 tx=0).stop 2023-10-30T17:32:39.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 10 monclient: shutdown 2023-10-30T17:32:39.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 -- 172.21.15.170:0/3889062015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57440aaed0 msgr2=0x7f574413f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57440aaed0 0x7f574413f9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f57340098c0 tx=0x7f57340046c0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 -- 172.21.15.170:0/3889062015 shutdown_connections 2023-10-30T17:32:39.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f573006c880 0x7f573006ed40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57440ab2b0 0x7f574406d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57440aaed0 0x7f574413f9f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 --2- 172.21.15.170:0/3889062015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57440a65a0 0x7f57441300f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 -- 172.21.15.170:0/3889062015 >> 172.21.15.170:0/3889062015 conn(0x7f57440b3110 msgr2=0x7f574414c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 -- 172.21.15.170:0/3889062015 shutdown_connections 2023-10-30T17:32:39.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 -- 172.21.15.170:0/3889062015 wait complete. 2023-10-30T17:32:39.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.625+0000 7f5749483640 1 librados: shutdown 2023-10-30T17:32:39.641 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476738 for osd.3 2023-10-30T17:32:39.747 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.746+0000 7f7772ffd640 1 -- 172.21.15.170:0/353202548 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7774007550 con 0x7f77840b7c90 2023-10-30T17:32:39.747 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.746+0000 7f7772ffd640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f776406c850 0x7f776406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.746+0000 7f7781d9b640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f776406c850 0x7f776406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.746+0000 7f7781d9b640 10 monclient: get_auth_request con 0x7f776406c850 auth_method 0 2023-10-30T17:32:39.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.747+0000 7f7772ffd640 10 client.4348.objecter ms_handle_connect 0x7f776406c850 2023-10-30T17:32:39.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.747+0000 7f7781d9b640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f776406c850 0x7f776406ed10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f778413c2d0 tx=0x7f7778008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.751+0000 7f7772ffd640 1 -- 172.21.15.170:0/353202548 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f77740584f0 con 0x7f77840b7c90 2023-10-30T17:32:39.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.751+0000 7f7772ffd640 10 client.4348.objecter ms_dispatch 0x7f7784000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:39.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.751+0000 7f7772ffd640 3 client.4348.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:39.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.751+0000 7f7772ffd640 3 client.4348.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:39.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.751+0000 7f7772ffd640 20 client.4348.objecter dump_active .. 0 homeless 2023-10-30T17:32:39.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.760+0000 7f7772ffd640 1 -- 172.21.15.170:0/353202548 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7774054130 con 0x7f77840b7c90 2023-10-30T17:32:39.762 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.760+0000 7f7772ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.762 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.760+0000 7f7772ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:39.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.777+0000 7efd5dffb640 1 -- 172.21.15.170:0/4126556675 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efd6001d3b0 con 0x7efd68145a20 2023-10-30T17:32:39.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.777+0000 7efd5dffb640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd4c06c850 0x7efd4c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.778+0000 7efd6d473640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd4c06c850 0x7efd4c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.780 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.778+0000 7efd6d473640 10 monclient: get_auth_request con 0x7efd4c06c850 auth_method 0 2023-10-30T17:32:39.780 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.778+0000 7efd5dffb640 10 client.4345.objecter ms_handle_connect 0x7efd4c06c850 2023-10-30T17:32:39.780 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.778+0000 7efd6d473640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd4c06c850 0x7efd4c06ed10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd64001550 tx=0x7efd64007a40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.781 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.779+0000 7efd5dffb640 1 -- 172.21.15.170:0/4126556675 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efd6000ce30 con 0x7efd68145a20 2023-10-30T17:32:39.781 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.779+0000 7efd5dffb640 10 client.4345.objecter ms_dispatch 0x7efd68000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:39.781 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.779+0000 7efd5dffb640 3 client.4345.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:39.781 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.779+0000 7efd5dffb640 3 client.4345.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:39.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.779+0000 7efd5dffb640 20 client.4345.objecter dump_active .. 0 homeless 2023-10-30T17:32:39.798 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.796+0000 7efd5dffb640 1 -- 172.21.15.170:0/4126556675 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efd60058350 con 0x7efd68145a20 2023-10-30T17:32:39.798 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.796+0000 7efd5dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.798 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.796+0000 7efd5dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:39.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.797+0000 7ff9c7fff640 1 -- 172.21.15.170:0/2086724736 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff9d8017020 con 0x7ff9e00b36a0 2023-10-30T17:32:39.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.797+0000 7ff9c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T17:32:39.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.797+0000 7ff9c7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:39.799 INFO:teuthology.orchestra.run.smithi170.stdout:73014444034 2023-10-30T17:32:39.801 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 10 librados: watch_flush enter 2023-10-30T17:32:39.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 10 librados: watch_flush exit 2023-10-30T17:32:39.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 20 client.4336.objecter shutdown clearing up homeless session... 2023-10-30T17:32:39.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 10 client.4336.objecter successfully canceled tick 2023-10-30T17:32:39.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 -- 172.21.15.170:0/2086724736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff9cc06c880 msgr2=0x7ff9cc06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff9cc06c880 0x7ff9cc06ed40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009670 tx=0x7ff9d4006bf0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.803 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 10 monclient: shutdown 2023-10-30T17:32:39.803 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.803 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 -- 172.21.15.170:0/2086724736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00b36a0 msgr2=0x7ff9e00adec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.803 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00b36a0 0x7ff9e00adec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c780 tx=0x7ff9d800cc40 comp rx=0 tx=0).stop 2023-10-30T17:32:39.803 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 -- 172.21.15.170:0/2086724736 shutdown_connections 2023-10-30T17:32:39.804 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff9cc06c880 0x7ff9cc06ed40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.804 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e00b36a0 0x7ff9e00adec0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.804 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9e00ac920 0x7ff9e013a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.804 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 --2- 172.21.15.170:0/2086724736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9e00a91d0 0x7ff9e0134030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.804 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 -- 172.21.15.170:0/2086724736 >> 172.21.15.170:0/2086724736 conn(0x7ff9e00afdf0 msgr2=0x7ff9e01c9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.805 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 -- 172.21.15.170:0/2086724736 shutdown_connections 2023-10-30T17:32:39.805 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 -- 172.21.15.170:0/2086724736 wait complete. 2023-10-30T17:32:39.805 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.800+0000 7ff9e4a21640 1 librados: shutdown 2023-10-30T17:32:39.812 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444034 for osd.0 2023-10-30T17:32:39.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.841+0000 7f6086da5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T17:32:39.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.841+0000 7f6086da5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T17:32:39.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.841+0000 7f6086da5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:39.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.841+0000 7f6086da5640 1 -- 172.21.15.170:0/1716984697 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f60800b3470 con 0x7f6080144f50 2023-10-30T17:32:39.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.846+0000 7f607d7fa640 1 -- 172.21.15.170:0/1716984697 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6078019070 con 0x7f6080144f50 2023-10-30T17:32:39.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.846+0000 7f607d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T17:32:39.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.846+0000 7f607d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:39.848 INFO:teuthology.orchestra.run.smithi170.stdout:68719476740 2023-10-30T17:32:39.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.849+0000 7f6086da5640 10 librados: watch_flush enter 2023-10-30T17:32:39.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.849+0000 7f6086da5640 10 librados: watch_flush exit 2023-10-30T17:32:39.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.849+0000 7f6086da5640 20 client.4341.objecter shutdown clearing up homeless session... 2023-10-30T17:32:39.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.849+0000 7f6086da5640 10 client.4341.objecter successfully canceled tick 2023-10-30T17:32:39.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.849+0000 7f6086da5640 1 -- 172.21.15.170:0/1716984697 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f606406c800 msgr2=0x7f606406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.849+0000 7f6086da5640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f606406c800 0x7f606406ecc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6074009710 tx=0x7f6074006c90 comp rx=0 tx=0).stop 2023-10-30T17:32:39.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.849+0000 7f6086da5640 10 monclient: shutdown 2023-10-30T17:32:39.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.849+0000 7f6086da5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.849+0000 7f6086da5640 1 -- 172.21.15.170:0/1716984697 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6080144f50 msgr2=0x7f60801d6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.849+0000 7f6086da5640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6080144f50 0x7f60801d6d30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f607800be20 tx=0x7f607800bf10 comp rx=0 tx=0).stop 2023-10-30T17:32:39.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.850+0000 7f6086da5640 1 -- 172.21.15.170:0/1716984697 shutdown_connections 2023-10-30T17:32:39.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.850+0000 7f6086da5640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f606406c800 0x7f606406ecc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.850+0000 7f6086da5640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6080144f50 0x7f60801d6d30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.850+0000 7f6086da5640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f608013eb60 0x7f6080143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.850+0000 7f6086da5640 1 --2- 172.21.15.170:0/1716984697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60800b7c90 0x7f60801428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.850+0000 7f6086da5640 1 -- 172.21.15.170:0/1716984697 >> 172.21.15.170:0/1716984697 conn(0x7f6080071da0 msgr2=0x7f60800b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.850+0000 7f6086da5640 1 -- 172.21.15.170:0/1716984697 shutdown_connections 2023-10-30T17:32:39.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.850+0000 7f6086da5640 1 -- 172.21.15.170:0/1716984697 wait complete. 2023-10-30T17:32:39.855 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.850+0000 7f6086da5640 1 librados: shutdown 2023-10-30T17:32:39.862 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476740 for osd.1 2023-10-30T17:32:39.922 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.919+0000 7f58ea7fc640 1 -- 172.21.15.170:0/3308378740 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f58e4004020 con 0x7f58f40a8b00 2023-10-30T17:32:39.922 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.920+0000 7f58ea7fc640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f58d806c800 0x7f58d806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.922 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.920+0000 7f58f98dd640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f58d806c800 0x7f58d806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.920+0000 7f58f98dd640 10 monclient: get_auth_request con 0x7f58d806c800 auth_method 0 2023-10-30T17:32:39.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.920+0000 7f58ea7fc640 10 client.4310.objecter ms_handle_connect 0x7f58d806c800 2023-10-30T17:32:39.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.920+0000 7f58f98dd640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f58d806c800 0x7f58d806ecc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009710 tx=0x7f58f0006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.935+0000 7f58ea7fc640 1 -- 172.21.15.170:0/3308378740 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f58e4059150 con 0x7f58f40a8b00 2023-10-30T17:32:39.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.935+0000 7f58ea7fc640 10 client.4310.objecter ms_dispatch 0x7f58f4000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:39.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.935+0000 7f58ea7fc640 3 client.4310.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:39.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.935+0000 7f58ea7fc640 3 client.4310.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:39.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.935+0000 7f58ea7fc640 20 client.4310.objecter dump_active .. 0 homeless 2023-10-30T17:32:39.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.935+0000 7f58ea7fc640 1 -- 172.21.15.170:0/3308378740 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f58e4054d90 con 0x7f58f40a8b00 2023-10-30T17:32:39.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.935+0000 7f58ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.935+0000 7f58ea7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:39.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.967+0000 7fc888ff9640 1 -- 172.21.15.170:0/2799804909 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc87c04d450 con 0x7fc88c0b7c90 2023-10-30T17:32:39.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.967+0000 7fc888ff9640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc87806c880 0x7fc87806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:39.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.968+0000 7fc88b7fe640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc87806c880 0x7fc87806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:39.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.968+0000 7fc88b7fe640 10 monclient: get_auth_request con 0x7fc87806c880 auth_method 0 2023-10-30T17:32:39.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.968+0000 7fc888ff9640 10 client.4313.objecter ms_handle_connect 0x7fc87806c880 2023-10-30T17:32:39.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.968+0000 7fc88b7fe640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc87806c880 0x7fc87806ed40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc88c144f20 tx=0x7fc880009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:39.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.973+0000 7facceffd640 1 -- 172.21.15.170:0/3901805742 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7facd80c49f0 con 0x7facd80bfc20 2023-10-30T17:32:39.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.973+0000 7facceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T17:32:39.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.973+0000 7facceffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:39.976 INFO:teuthology.orchestra.run.smithi170.stdout:60129542147 2023-10-30T17:32:39.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 10 librados: watch_flush enter 2023-10-30T17:32:39.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 10 librados: watch_flush exit 2023-10-30T17:32:39.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 20 client.4298.objecter shutdown clearing up homeless session... 2023-10-30T17:32:39.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 10 client.4298.objecter successfully canceled tick 2023-10-30T17:32:39.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 -- 172.21.15.170:0/3901805742 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7facc006c850 msgr2=0x7facc006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7facc006c850 0x7facc006ed10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facd40017f0 tx=0x7facd4009040 comp rx=0 tx=0).stop 2023-10-30T17:32:39.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 10 monclient: shutdown 2023-10-30T17:32:39.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 -- 172.21.15.170:0/3901805742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 msgr2=0x7facd815d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd815d640 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7face0061be0 tx=0x7face0000b00 comp rx=0 tx=0).stop 2023-10-30T17:32:39.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 -- 172.21.15.170:0/3901805742 shutdown_connections 2023-10-30T17:32:39.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7facc006c850 0x7facc006ed10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd8035180 0x7facd815d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facd80bfc20 0x7facd815d640 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 --2- 172.21.15.170:0/3901805742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facd8034da0 0x7facd80c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 -- 172.21.15.170:0/3901805742 >> 172.21.15.170:0/3901805742 conn(0x7facd80061f0 msgr2=0x7facd80bf2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 -- 172.21.15.170:0/3901805742 shutdown_connections 2023-10-30T17:32:39.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.975+0000 7face4cf2640 1 -- 172.21.15.170:0/3901805742 wait complete. 2023-10-30T17:32:39.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.976+0000 7face4cf2640 1 librados: shutdown 2023-10-30T17:32:39.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.983+0000 7f7788851640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T17:32:39.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.983+0000 7f7788851640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T17:32:39.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.983+0000 7f7788851640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:39.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.983+0000 7f7788851640 1 -- 172.21.15.170:0/353202548 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f77841d62d0 con 0x7f77840b7c90 2023-10-30T17:32:39.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.987+0000 7f7772ffd640 1 -- 172.21.15.170:0/353202548 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7774057750 con 0x7f77840b7c90 2023-10-30T17:32:39.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.987+0000 7f7772ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T17:32:39.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.987+0000 7f7772ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:39.990 INFO:teuthology.orchestra.run.smithi170.stdout:64424509443 2023-10-30T17:32:39.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 10 librados: watch_flush enter 2023-10-30T17:32:39.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 10 librados: watch_flush exit 2023-10-30T17:32:39.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 20 client.4348.objecter shutdown clearing up homeless session... 2023-10-30T17:32:39.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 10 client.4348.objecter successfully canceled tick 2023-10-30T17:32:39.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 -- 172.21.15.170:0/353202548 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f776406c850 msgr2=0x7f776406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f776406c850 0x7f776406ed10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f778413c2d0 tx=0x7f7778008040 comp rx=0 tx=0).stop 2023-10-30T17:32:39.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 10 monclient: shutdown 2023-10-30T17:32:39.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:39.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 -- 172.21.15.170:0/353202548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77840b7c90 msgr2=0x7f7784144540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:39.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77840b7c90 0x7f7784144540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f777400c500 tx=0x7f777400c9c0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 -- 172.21.15.170:0/353202548 shutdown_connections 2023-10-30T17:32:39.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f776406c850 0x7f776406ed10 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f778413c2d0 tx=0x7f7778008040 comp rx=0 tx=0).stop 2023-10-30T17:32:39.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7784144f50 0x7f77841d65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f778413eb60 0x7f77841438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 --2- 172.21.15.170:0/353202548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77840b7c90 0x7f7784144540 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:39.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 -- 172.21.15.170:0/353202548 >> 172.21.15.170:0/353202548 conn(0x7f7784071da0 msgr2=0x7f77840b72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:39.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.989+0000 7f7788851640 1 -- 172.21.15.170:0/353202548 shutdown_connections 2023-10-30T17:32:39.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.990+0000 7f7788851640 1 -- 172.21.15.170:0/353202548 wait complete. 2023-10-30T17:32:39.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.990+0000 7f7788851640 1 librados: shutdown 2023-10-30T17:32:39.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.990+0000 7fc888ff9640 1 -- 172.21.15.170:0/2799804909 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc87c02d040 con 0x7fc88c0b7c90 2023-10-30T17:32:39.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.990+0000 7fc888ff9640 10 client.4313.objecter ms_dispatch 0x7fc88c000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:39.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.990+0000 7fc888ff9640 3 client.4313.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:39.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.990+0000 7fc888ff9640 3 client.4313.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:39.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.990+0000 7fc888ff9640 20 client.4313.objecter dump_active .. 0 homeless 2023-10-30T17:32:39.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.990+0000 7fc888ff9640 1 -- 172.21.15.170:0/2799804909 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc87c077120 con 0x7fc88c0b7c90 2023-10-30T17:32:39.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.990+0000 7fc888ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:39.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:39.990+0000 7fc888ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:39.999 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.7 2023-10-30T17:32:40.005 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.2 2023-10-30T17:32:40.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.068+0000 7efd6f6d7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T17:32:40.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.068+0000 7efd6f6d7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T17:32:40.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.068+0000 7efd6f6d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:40.072 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.068+0000 7efd6f6d7640 1 -- 172.21.15.170:0/4126556675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7efd681db2e0 con 0x7efd68145a20 2023-10-30T17:32:40.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.077+0000 7efd5dffb640 1 -- 172.21.15.170:0/4126556675 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd681db2e0 con 0x7efd68145a20 2023-10-30T17:32:40.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.077+0000 7efd5dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T17:32:40.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.077+0000 7efd5dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:40.080 INFO:teuthology.orchestra.run.smithi170.stdout:60129542151 2023-10-30T17:32:40.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.080+0000 7efd6f6d7640 10 librados: watch_flush enter 2023-10-30T17:32:40.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.080+0000 7efd6f6d7640 10 librados: watch_flush exit 2023-10-30T17:32:40.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.080+0000 7efd6f6d7640 20 client.4345.objecter shutdown clearing up homeless session... 2023-10-30T17:32:40.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.080+0000 7efd6f6d7640 10 client.4345.objecter successfully canceled tick 2023-10-30T17:32:40.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.080+0000 7efd6f6d7640 1 -- 172.21.15.170:0/4126556675 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd4c06c850 msgr2=0x7efd4c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.080+0000 7efd6f6d7640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd4c06c850 0x7efd4c06ed10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd64001550 tx=0x7efd64007a40 comp rx=0 tx=0).stop 2023-10-30T17:32:40.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.080+0000 7efd6f6d7640 10 monclient: shutdown 2023-10-30T17:32:40.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.080+0000 7efd6f6d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:40.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.080+0000 7efd6f6d7640 1 -- 172.21.15.170:0/4126556675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd68145a20 msgr2=0x7efd680afbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.080+0000 7efd6f6d7640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd68145a20 0x7efd680afbf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efd6000b970 tx=0x7efd6000be30 comp rx=0 tx=0).stop 2023-10-30T17:32:40.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.081+0000 7efd6f6d7640 1 -- 172.21.15.170:0/4126556675 shutdown_connections 2023-10-30T17:32:40.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.081+0000 7efd6f6d7640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd4c06c850 0x7efd4c06ed10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.081+0000 7efd6f6d7640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd68145a20 0x7efd680afbf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.081+0000 7efd6f6d7640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd680a8b00 0x7efd681db9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.081+0000 7efd6f6d7640 1 --2- 172.21.15.170:0/4126556675 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd680a8720 0x7efd68138520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.081+0000 7efd6f6d7640 1 -- 172.21.15.170:0/4126556675 >> 172.21.15.170:0/4126556675 conn(0x7efd680b7300 msgr2=0x7efd6813cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:40.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.081+0000 7efd6f6d7640 1 -- 172.21.15.170:0/4126556675 shutdown_connections 2023-10-30T17:32:40.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.081+0000 7efd6f6d7640 1 -- 172.21.15.170:0/4126556675 wait complete. 2023-10-30T17:32:40.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.081+0000 7efd6f6d7640 1 librados: shutdown 2023-10-30T17:32:40.099 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.4 2023-10-30T17:32:40.100 DEBUG:teuthology.parallel:result is None 2023-10-30T17:32:40.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.186+0000 7fc8925c9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T17:32:40.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.186+0000 7fc8925c9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T17:32:40.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.186+0000 7fc8925c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:40.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.186+0000 7fc8925c9640 1 -- 172.21.15.170:0/2799804909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc88c1d9c40 con 0x7fc88c0b7c90 2023-10-30T17:32:40.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.192+0000 7fc888ff9640 1 -- 172.21.15.170:0/2799804909 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc88c1d9c40 con 0x7fc88c0b7c90 2023-10-30T17:32:40.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.192+0000 7fc888ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T17:32:40.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.192+0000 7fc888ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:40.195 INFO:teuthology.orchestra.run.smithi170.stdout:55834574854 2023-10-30T17:32:40.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 10 librados: watch_flush enter 2023-10-30T17:32:40.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 10 librados: watch_flush exit 2023-10-30T17:32:40.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 20 client.4313.objecter shutdown clearing up homeless session... 2023-10-30T17:32:40.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 10 client.4313.objecter successfully canceled tick 2023-10-30T17:32:40.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 -- 172.21.15.170:0/2799804909 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc87806c880 msgr2=0x7fc87806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc87806c880 0x7fc87806ed40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc88c144f20 tx=0x7fc880009040 comp rx=0 tx=0).stop 2023-10-30T17:32:40.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 10 monclient: shutdown 2023-10-30T17:32:40.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:40.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 -- 172.21.15.170:0/2799804909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c0b7c90 msgr2=0x7fc88c13a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c0b7c90 0x7fc88c13a6e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc87c033000 tx=0x7fc87c0046c0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 -- 172.21.15.170:0/2799804909 shutdown_connections 2023-10-30T17:32:40.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc87806c880 0x7fc87806ed40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc88c13eb60 0x7fc88c13ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c144f50 0x7fc88c135470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 --2- 172.21.15.170:0/2799804909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c0b7c90 0x7fc88c13a6e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 -- 172.21.15.170:0/2799804909 >> 172.21.15.170:0/2799804909 conn(0x7fc88c071da0 msgr2=0x7fc88c0b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:40.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 -- 172.21.15.170:0/2799804909 shutdown_connections 2023-10-30T17:32:40.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.195+0000 7fc8925c9640 1 -- 172.21.15.170:0/2799804909 wait complete. 2023-10-30T17:32:40.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.196+0000 7fc8925c9640 1 librados: shutdown 2023-10-30T17:32:40.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.198+0000 7f58fbb41640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T17:32:40.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.198+0000 7f58fbb41640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T17:32:40.203 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.198+0000 7f58fbb41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:40.203 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.198+0000 7f58fbb41640 1 -- 172.21.15.170:0/3308378740 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f58f40ac200 con 0x7f58f40a8b00 2023-10-30T17:32:40.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.203+0000 7f58ea7fc640 1 -- 172.21.15.170:0/3308378740 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f58e401f070 con 0x7f58f40a8b00 2023-10-30T17:32:40.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.203+0000 7f58ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-10-30T17:32:40.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.203+0000 7f58ea7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:40.206 INFO:teuthology.orchestra.run.smithi170.stdout:60129542150 2023-10-30T17:32:40.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 10 librados: watch_flush enter 2023-10-30T17:32:40.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 10 librados: watch_flush exit 2023-10-30T17:32:40.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 20 client.4310.objecter shutdown clearing up homeless session... 2023-10-30T17:32:40.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 10 client.4310.objecter successfully canceled tick 2023-10-30T17:32:40.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 1 -- 172.21.15.170:0/3308378740 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f58d806c800 msgr2=0x7f58d806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f58d806c800 0x7f58d806ecc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009710 tx=0x7f58f0006c90 comp rx=0 tx=0).stop 2023-10-30T17:32:40.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 10 monclient: shutdown 2023-10-30T17:32:40.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:40.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 1 -- 172.21.15.170:0/3308378740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f40a8b00 msgr2=0x7f58f40b5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f40a8b00 0x7f58f40b5a40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58e400c7f0 tx=0x7f58e400ccb0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 1 -- 172.21.15.170:0/3308378740 shutdown_connections 2023-10-30T17:32:40.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f58d806c800 0x7f58d806ecc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58f413f4d0 0x7f58f40ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58f40a8b00 0x7f58f40b5a40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.205+0000 7f58fbb41640 1 --2- 172.21.15.170:0/3308378740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f40a8720 0x7f58f40b3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.206+0000 7f58fbb41640 1 -- 172.21.15.170:0/3308378740 >> 172.21.15.170:0/3308378740 conn(0x7f58f40b7300 msgr2=0x7f58f40b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:40.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.206+0000 7f58fbb41640 1 -- 172.21.15.170:0/3308378740 shutdown_connections 2023-10-30T17:32:40.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.206+0000 7f58fbb41640 1 -- 172.21.15.170:0/3308378740 wait complete. 2023-10-30T17:32:40.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.206+0000 7f58fbb41640 1 librados: shutdown 2023-10-30T17:32:40.213 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2023-10-30T17:32:40.213 DEBUG:teuthology.parallel:result is None 2023-10-30T17:32:40.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.6 2023-10-30T17:32:40.219 DEBUG:teuthology.parallel:result is None 2023-10-30T17:32:40.642 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T17:32:40.746 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.743+0000 7f708f56c640 10 monclient: get_monmap_and_config 2023-10-30T17:32:40.747 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.743+0000 7f708f56c640 10 monclient: build_initial_monmap 2023-10-30T17:32:40.747 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.743+0000 7f708f56c640 10 monclient: monmap: 2023-10-30T17:32:40.747 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:40.747 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.747 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:40.745625+0000 2023-10-30T17:32:40.748 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:40.745625+0000 2023-10-30T17:32:40.748 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:40.748 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.748 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:40.748 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:40.749 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:40.749 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:40.749 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:40.749 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.744+0000 7f708f56c640 1 Processor -- start 2023-10-30T17:32:40.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.744+0000 7f708f56c640 1 -- start start 2023-10-30T17:32:40.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.744+0000 7f708f56c640 10 monclient: init 2023-10-30T17:32:40.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.744+0000 7f708f56c640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:40.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.744+0000 7f708f56c640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-10-30T17:32:40.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.744+0000 7f708f56c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70880b4530 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:40.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.744+0000 7f708f56c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70880a84b0 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:40.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.744+0000 7f708f56c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7088145680 0x7f708814d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.744+0000 7f708f56c640 10 monclient(hunting): picked mon.noname-a con 0x7f7088145680 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:40.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.744+0000 7f708f56c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70880a8c80 con 0x7f70880b4530 2023-10-30T17:32:40.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.745+0000 7f708f56c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70880aa4b0 con 0x7f70880a84b0 2023-10-30T17:32:40.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.745+0000 7f708f56c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.745+0000 7f708f56c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70880b4910 con 0x7f7088145680 2023-10-30T17:32:40.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.745+0000 7f708f56c640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:40.755 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.745+0000 7f708f56c640 10 monclient(hunting): authenticate will time out at 1343.069082s 2023-10-30T17:32:40.755 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.745+0000 7f708db09640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7088145680 0x7f708814d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:40.755 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.745+0000 7f708cb07640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f70880a84b0 0x7f7088144f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:46908/0 (socket says 172.21.15.170:46908) 2023-10-30T17:32:40.755 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.745+0000 7f708cb07640 1 -- 172.21.15.170:0/4073573938 learned_addr learned my addr 172.21.15.170:0/4073573938 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:40.755 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.746+0000 7f708db09640 10 monclient(hunting): get_auth_request con 0x7f7088145680 auth_method 0 2023-10-30T17:32:40.756 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.746+0000 7f708db09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:40.756 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.746+0000 7f708db09640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:40.756 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.746+0000 7f708db09640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:40.756 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.746+0000 7f708db09640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:40.756 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.746+0000 7f708db09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:40.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.746+0000 7f708db09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:40.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f707ffff640 1 -- 172.21.15.170:0/4073573938 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920992244 0 0) 0x7f70880aa4b0 con 0x7f70880a84b0 2023-10-30T17:32:40.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f707ffff640 10 monclient(hunting): _negotiate 2023-10-30T17:32:40.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f707ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:40.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f707ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:40.758 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f707ffff640 10 monclient(hunting): my global_id is 4354 2023-10-30T17:32:40.758 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f707ffff640 1 -- 172.21.15.170:0/4073573938 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7074003520 con 0x7f70880a84b0 2023-10-30T17:32:40.758 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f708db09640 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2023-10-30T17:32:40.758 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f708db09640 1 -- 172.21.15.170:0/4073573938 >> v1:172.21.15.177:6789/0 conn(0x7f70880a84b0 legacy=0x7f7088144f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.759 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f708db09640 1 -- 172.21.15.170:0/4073573938 >> v1:172.21.15.170:6789/0 conn(0x7f70880b4530 legacy=0x7f708813eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.759 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f708db09640 10 monclient: _finish_hunting 0 2023-10-30T17:32:40.759 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f708db09640 1 monclient: found mon.noname-a 2023-10-30T17:32:40.759 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f708db09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:40.759 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f708db09640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:40.760 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.748+0000 7f708db09640 1 -- 172.21.15.170:0/4073573938 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70880aa790 con 0x7f7088145680 2023-10-30T17:32:40.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.752+0000 7f708db09640 1 --2- 172.21.15.170:0/4073573938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7088145680 0x7f708814d2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7080009d90 tx=0x7f708002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb264571eed9e29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:40.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.755+0000 7f707ffff640 1 -- 172.21.15.170:0/4073573938 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708002cd00 con 0x7f7088145680 2023-10-30T17:32:40.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.755+0000 7f707ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:40.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.755+0000 7f707ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:40.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.755+0000 7f707ffff640 10 monclient: dump: 2023-10-30T17:32:40.765 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:40.765 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.765 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.766 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.766 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:40.766 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.766 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:40.767 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:40.767 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:40.767 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.755+0000 7f707ffff640 10 monclient: _finish_auth 0 2023-10-30T17:32:40.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.755+0000 7f707ffff640 10 monclient: _check_auth_tickets 2023-10-30T17:32:40.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.755+0000 7f707ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:40.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.755+0000 7f708f56c640 5 monclient: authenticate success, global_id 4344 2023-10-30T17:32:40.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.755+0000 7f708f56c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:40.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.756+0000 7f707ffff640 1 -- 172.21.15.170:0/4073573938 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708002ce60 con 0x7f7088145680 2023-10-30T17:32:40.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.756+0000 7f707ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:40.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.756+0000 7f707ffff640 1 -- 172.21.15.170:0/4073573938 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080034640 con 0x7f7088145680 2023-10-30T17:32:40.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.756+0000 7f707ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:40.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.756+0000 7f707ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:40.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.756+0000 7f707ffff640 10 monclient: dump: 2023-10-30T17:32:40.769 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:40.770 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.770 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.770 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.770 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:40.770 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.771 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:40.771 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:40.771 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:40.771 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.756+0000 7f708f56c640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:40.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.756+0000 7f708f56c640 4 set_mon_vals no callback set 2023-10-30T17:32:40.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.757+0000 7f708f56c640 10 monclient: shutdown 2023-10-30T17:32:40.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.757+0000 7f708f56c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:40.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.757+0000 7f708f56c640 1 -- 172.21.15.170:0/4073573938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7088145680 msgr2=0x7f708814d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.757+0000 7f708f56c640 1 --2- 172.21.15.170:0/4073573938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7088145680 0x7f708814d2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7080009d90 tx=0x7f708002bf50 comp rx=0 tx=0).stop 2023-10-30T17:32:40.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.757+0000 7f708f56c640 1 -- 172.21.15.170:0/4073573938 shutdown_connections 2023-10-30T17:32:40.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.757+0000 7f708f56c640 1 --2- 172.21.15.170:0/4073573938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7088145680 0x7f708814d2d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.757+0000 7f708f56c640 1 -- 172.21.15.170:0/4073573938 >> 172.21.15.170:0/4073573938 conn(0x7f7088071b90 msgr2=0x7f7088071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:40.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.757+0000 7f708f56c640 1 -- 172.21.15.170:0/4073573938 shutdown_connections 2023-10-30T17:32:40.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.757+0000 7f708f56c640 1 -- 172.21.15.170:0/4073573938 wait complete. 2023-10-30T17:32:40.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.758+0000 7f708f56c640 10 monclient: build_initial_monmap 2023-10-30T17:32:40.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.758+0000 7f708f56c640 10 monclient: monmap: 2023-10-30T17:32:40.774 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:40.774 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:40.775 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:40.775 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:40.775 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:40.775 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.776 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:40.776 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:40.776 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:40.776 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.758+0000 7f708f56c640 1 librados: starting msgr at 2023-10-30T17:32:40.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.758+0000 7f708f56c640 1 librados: starting objecter 2023-10-30T17:32:40.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 1 Processor -- start 2023-10-30T17:32:40.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 1 -- start start 2023-10-30T17:32:40.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 1 librados: setting wanted keys 2023-10-30T17:32:40.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 1 librados: calling monclient init 2023-10-30T17:32:40.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient: init 2023-10-30T17:32:40.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:40.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:32:40.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70880b4530 0x7f70880b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient(hunting): picked mon.noname-c con 0x7f70880b4530 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:40.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7088145680 0x7f70880b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient(hunting): picked mon.noname-b con 0x7f7088145680 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:40.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7088138470 0x7f70881388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient(hunting): picked mon.noname-a con 0x7f7088138470 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:40.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.780 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70880aba60 con 0x7f7088138470 2023-10-30T17:32:40.780 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.780 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70880b6370 con 0x7f7088145680 2023-10-30T17:32:40.780 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.781 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70880ad7b0 con 0x7f70880b4530 2023-10-30T17:32:40.781 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:40.781 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.759+0000 7f708f56c640 10 monclient(hunting): authenticate will time out at 1343.083722s 2023-10-30T17:32:40.781 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708db09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7088138470 0x7f70881388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:40.781 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708d308640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70880b4530 0x7f70880b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:40.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708cb07640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7088145680 0x7f70880b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:40.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708d308640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70880b4530 0x7f70880b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:35356/0 (socket says 172.21.15.170:35356) 2023-10-30T17:32:40.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708db09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7088138470 0x7f70881388d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:58650/0 (socket says 172.21.15.170:58650) 2023-10-30T17:32:40.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708d308640 1 -- 172.21.15.170:0/1232012392 learned_addr learned my addr 172.21.15.170:0/1232012392 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:40.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708d308640 10 monclient(hunting): get_auth_request con 0x7f70880b4530 auth_method 0 2023-10-30T17:32:40.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708d308640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:40.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708d308640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:40.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708d308640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:40.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708db09640 10 monclient(hunting): get_auth_request con 0x7f7088138470 auth_method 0 2023-10-30T17:32:40.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708db09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:40.784 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708db09640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:40.784 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708db09640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:40.784 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708cb07640 10 monclient(hunting): get_auth_request con 0x7f7088145680 auth_method 0 2023-10-30T17:32:40.784 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708cb07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:40.784 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708cb07640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:40.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.760+0000 7f708cb07640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:40.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.761+0000 7f708db09640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:40.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.761+0000 7f708db09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:40.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.761+0000 7f708db09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:40.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.761+0000 7f708d308640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:40.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.761+0000 7f708d308640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:40.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.761+0000 7f708d308640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:40.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.761+0000 7f708cb07640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:40.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.761+0000 7f708cb07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:40.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.761+0000 7f708cb07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:40.788 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.763+0000 7f708cb07640 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2023-10-30T17:32:40.788 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.763+0000 7f708cb07640 1 -- 172.21.15.170:0/1232012392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70880b4530 msgr2=0x7f70880b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.788 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.763+0000 7f708cb07640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70880b4530 0x7f70880b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.763+0000 7f708cb07640 1 -- 172.21.15.170:0/1232012392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7088138470 msgr2=0x7f70881388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.763+0000 7f708cb07640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7088138470 0x7f70881388d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.763+0000 7f708cb07640 10 monclient: _finish_hunting 0 2023-10-30T17:32:40.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.763+0000 7f708cb07640 1 monclient: found mon.noname-b 2023-10-30T17:32:40.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.763+0000 7f708cb07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:40.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.763+0000 7f708cb07640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:40.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.763+0000 7f708cb07640 1 -- 172.21.15.170:0/1232012392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7080009a70 con 0x7f7088145680 2023-10-30T17:32:40.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.766+0000 7f708cb07640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7088145680 0x7f70880b5e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f707800c3c0 tx=0x7f7078001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:40.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.766+0000 7f707dffb640 10 client.?.objecter ms_handle_connect 0x7f7088145680 2023-10-30T17:32:40.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.766+0000 7f707dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:40.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.769+0000 7f707dffb640 1 -- 172.21.15.170:0/1232012392 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078005ce0 con 0x7f7088145680 2023-10-30T17:32:40.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.769+0000 7f707dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:40.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.769+0000 7f707dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:40.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.769+0000 7f707dffb640 10 monclient: dump: 2023-10-30T17:32:40.791 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:40.792 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.792 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.792 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.792 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:40.792 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.793 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:40.793 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:40.793 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:40.793 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.769+0000 7f707dffb640 10 monclient: _finish_auth 0 2023-10-30T17:32:40.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.769+0000 7f707dffb640 10 monclient: _check_auth_tickets 2023-10-30T17:32:40.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.769+0000 7f707dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:40.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.769+0000 7f707dffb640 1 -- 172.21.15.170:0/1232012392 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7078005e40 con 0x7f7088145680 2023-10-30T17:32:40.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.769+0000 7f707dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:40.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f708f56c640 5 monclient: authenticate success, global_id 4357 2023-10-30T17:32:40.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f707dffb640 1 -- 172.21.15.170:0/1232012392 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70780072d0 con 0x7f7088145680 2023-10-30T17:32:40.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f708f56c640 10 monclient: _renew_subs 2023-10-30T17:32:40.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f708f56c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:40.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f708f56c640 1 -- 172.21.15.170:0/1232012392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7088138e10 con 0x7f7088145680 2023-10-30T17:32:40.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f707f7fe640 4 set_mon_vals no callback set 2023-10-30T17:32:40.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f707dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:40.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f707dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:40.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f707dffb640 10 monclient: dump: 2023-10-30T17:32:40.796 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:40.796 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.797 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.797 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.797 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:40.797 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.797 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:40.798 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:40.798 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:40.798 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.798 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f708f56c640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:40.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f708f56c640 10 monclient: _renew_subs 2023-10-30T17:32:40.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f708f56c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:40.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f708f56c640 1 -- 172.21.15.170:0/1232012392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70880ac140 con 0x7f7088145680 2023-10-30T17:32:40.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.770+0000 7f708f56c640 1 librados: init done 2023-10-30T17:32:40.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.771+0000 7f708f56c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:40.800 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.771+0000 7f708f56c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:40.800 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.771+0000 7f708f56c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:40.800 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.771+0000 7f708f56c640 1 -- 172.21.15.170:0/1232012392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70880b7bc0 con 0x7f7088145680 2023-10-30T17:32:40.801 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.788+0000 7f707dffb640 1 -- 172.21.15.170:0/1232012392 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7078007430 con 0x7f7088145680 2023-10-30T17:32:40.801 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.788+0000 7f707dffb640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f706c06c880 0x7f706c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.789+0000 7f708d308640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f706c06c880 0x7f706c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:40.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.789+0000 7f708d308640 10 monclient: get_auth_request con 0x7f706c06c880 auth_method 0 2023-10-30T17:32:40.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.789+0000 7f707dffb640 10 client.4357.objecter ms_handle_connect 0x7f706c06c880 2023-10-30T17:32:40.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.789+0000 7f708d308640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f706c06c880 0x7f706c06ed40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7084001840 tx=0x7f7084009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:40.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.791+0000 7f707dffb640 1 -- 172.21.15.170:0/1232012392 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7078057df0 con 0x7f7088145680 2023-10-30T17:32:40.803 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.791+0000 7f707dffb640 10 client.4357.objecter ms_dispatch 0x7f7088000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:40.803 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.791+0000 7f707dffb640 3 client.4357.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:40.803 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.791+0000 7f707dffb640 3 client.4357.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:40.803 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.791+0000 7f707dffb640 20 client.4357.objecter dump_active .. 0 homeless 2023-10-30T17:32:40.813 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T17:32:40.863 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T17:32:40.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.913+0000 7f44df8b4640 10 monclient: get_monmap_and_config 2023-10-30T17:32:40.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.913+0000 7f44df8b4640 10 monclient: build_initial_monmap 2023-10-30T17:32:40.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.913+0000 7f44df8b4640 10 monclient: monmap: 2023-10-30T17:32:40.916 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:40.917 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.917 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:40.915651+0000 2023-10-30T17:32:40.917 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:40.915651+0000 2023-10-30T17:32:40.917 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:40.918 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.918 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:40.918 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:40.918 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:40.918 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:40.919 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:40.919 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.921 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.914+0000 7f44df8b4640 1 Processor -- start 2023-10-30T17:32:40.921 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.914+0000 7f44df8b4640 1 -- start start 2023-10-30T17:32:40.921 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.914+0000 7f44df8b4640 10 monclient: init 2023-10-30T17:32:40.921 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:40.922 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-10-30T17:32:40.922 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44d80b7c90 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:40.922 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44d813eb60 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:40.922 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d8145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 10 monclient(hunting): picked mon.noname-b con 0x7f44d8144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:40.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d80aa740 con 0x7f44d813eb60 2023-10-30T17:32:40.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d80aa9b0 con 0x7f44d80b7c90 2023-10-30T17:32:40.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d80a9500 con 0x7f44d8144f50 2023-10-30T17:32:40.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:40.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44df8b4640 10 monclient(hunting): authenticate will time out at 1343.239229s 2023-10-30T17:32:40.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44dde51640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d8145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:40.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44dce4f640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f44d813eb60 0x7f44d813efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:56796/0 (socket says 172.21.15.170:56796) 2023-10-30T17:32:40.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44dce4f640 1 -- 172.21.15.170:0/601866310 learned_addr learned my addr 172.21.15.170:0/601866310 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:40.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44dde51640 10 monclient(hunting): get_auth_request con 0x7f44d8144f50 auth_method 0 2023-10-30T17:32:40.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44dde51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:40.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44dde51640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:40.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.915+0000 7f44dde51640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:40.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.916+0000 7f44dde51640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:40.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.916+0000 7f44dde51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:40.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.916+0000 7f44dde51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:40.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.918+0000 7f44dde51640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2023-10-30T17:32:40.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.918+0000 7f44dde51640 1 -- 172.21.15.170:0/601866310 >> v1:172.21.15.177:6789/0 conn(0x7f44d80b7c90 legacy=0x7f44d80a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:40.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.918+0000 7f44dde51640 1 -- 172.21.15.170:0/601866310 >> v1:172.21.15.170:6789/0 conn(0x7f44d813eb60 legacy=0x7f44d813efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.918+0000 7f44dde51640 10 monclient: _finish_hunting 0 2023-10-30T17:32:40.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.918+0000 7f44dde51640 1 monclient: found mon.noname-b 2023-10-30T17:32:40.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.918+0000 7f44dde51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:40.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.918+0000 7f44dde51640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:40.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.918+0000 7f44dde51640 1 -- 172.21.15.170:0/601866310 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d80a9900 con 0x7f44d8144f50 2023-10-30T17:32:40.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.920+0000 7f44dde51640 1 --2- 172.21.15.170:0/601866310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d8145400 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009dc0 tx=0x7f44d002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b24b578017b36f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:40.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 1 -- 172.21.15.170:0/601866310 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d002cbb0 con 0x7f44d8144f50 2023-10-30T17:32:40.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:40.932 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:40.932 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 10 monclient: dump: 2023-10-30T17:32:40.932 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:40.932 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.932 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.933 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.933 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:40.933 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.933 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:40.933 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:40.934 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:40.934 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.934 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 10 monclient: _finish_auth 0 2023-10-30T17:32:40.934 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 10 monclient: _check_auth_tickets 2023-10-30T17:32:40.934 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:40.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 1 -- 172.21.15.170:0/601866310 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d002cd10 con 0x7f44d8144f50 2023-10-30T17:32:40.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 5 monclient: authenticate success, global_id 4319 2023-10-30T17:32:40.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:40.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:40.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:40.936 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 4 set_mon_vals no callback set 2023-10-30T17:32:40.936 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 1 -- 172.21.15.170:0/601866310 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0034660 con 0x7f44d8144f50 2023-10-30T17:32:40.936 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:40.936 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:40.936 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44cffff640 10 monclient: dump: 2023-10-30T17:32:40.937 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:40.937 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.937 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.937 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.938 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:40.938 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.938 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:40.938 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:40.939 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 10 monclient: shutdown 2023-10-30T17:32:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:40.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 1 -- 172.21.15.170:0/601866310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 msgr2=0x7f44d8145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 1 --2- 172.21.15.170:0/601866310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d8145400 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009dc0 tx=0x7f44d002be00 comp rx=0 tx=0).stop 2023-10-30T17:32:40.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 1 -- 172.21.15.170:0/601866310 shutdown_connections 2023-10-30T17:32:40.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 1 --2- 172.21.15.170:0/601866310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d8145400 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.924+0000 7f44df8b4640 1 -- 172.21.15.170:0/601866310 >> 172.21.15.170:0/601866310 conn(0x7f44d8071da0 msgr2=0x7f44d80b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:40.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.925+0000 7f44df8b4640 1 -- 172.21.15.170:0/601866310 shutdown_connections 2023-10-30T17:32:40.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.925+0000 7f44df8b4640 1 -- 172.21.15.170:0/601866310 wait complete. 2023-10-30T17:32:40.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.925+0000 7f44df8b4640 10 monclient: build_initial_monmap 2023-10-30T17:32:40.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.925+0000 7f44df8b4640 10 monclient: monmap: 2023-10-30T17:32:40.943 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:40.943 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:40.944 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:40.944 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:40.944 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:40.944 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.944 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:40.945 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:40.945 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:40.945 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 librados: starting msgr at 2023-10-30T17:32:40.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 librados: starting objecter 2023-10-30T17:32:40.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 Processor -- start 2023-10-30T17:32:40.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 -- start start 2023-10-30T17:32:40.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 librados: setting wanted keys 2023-10-30T17:32:40.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 librados: calling monclient init 2023-10-30T17:32:40.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient: init 2023-10-30T17:32:40.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:40.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:32:40.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44d80b7c90 0x7f44d813fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient(hunting): picked mon.noname-a con 0x7f44d80b7c90 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:40.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44d813eb60 0x7f44d8133c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient(hunting): picked mon.noname-b con 0x7f44d813eb60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:40.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d812e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient(hunting): picked mon.noname-c con 0x7f44d8144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:40.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8141360 con 0x7f44d80b7c90 2023-10-30T17:32:40.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8144c30 con 0x7f44d813eb60 2023-10-30T17:32:40.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d81429d0 con 0x7f44d8144f50 2023-10-30T17:32:40.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:40.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.926+0000 7f44df8b4640 10 monclient(hunting): authenticate will time out at 1343.250879s 2023-10-30T17:32:40.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dde51640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d812e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:40.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dd650640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44d80b7c90 0x7f44d813fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:40.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dde51640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d812e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:35376/0 (socket says 172.21.15.170:35376) 2023-10-30T17:32:40.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dde51640 1 -- 172.21.15.170:0/3333439696 learned_addr learned my addr 172.21.15.170:0/3333439696 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:40.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dde51640 10 monclient(hunting): get_auth_request con 0x7f44d8144f50 auth_method 0 2023-10-30T17:32:40.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dde51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:40.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dde51640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:40.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dde51640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:40.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dd650640 10 monclient(hunting): get_auth_request con 0x7f44d80b7c90 auth_method 0 2023-10-30T17:32:40.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dd650640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:40.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dd650640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:40.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.927+0000 7f44dd650640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:40.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.928+0000 7f44dde51640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:40.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.928+0000 7f44dde51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:40.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.928+0000 7f44dde51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:40.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.928+0000 7f44dd650640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:40.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.928+0000 7f44dd650640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:40.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.928+0000 7f44dd650640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:40.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.930+0000 7f44dde51640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2023-10-30T17:32:40.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.931+0000 7f44dde51640 1 -- 172.21.15.170:0/3333439696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44d813eb60 msgr2=0x7f44d8133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.931+0000 7f44dde51640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44d813eb60 0x7f44d8133c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.931+0000 7f44dde51640 1 -- 172.21.15.170:0/3333439696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44d80b7c90 msgr2=0x7f44d813fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.931+0000 7f44dde51640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44d80b7c90 0x7f44d813fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.931+0000 7f44dde51640 10 monclient: _finish_hunting 0 2023-10-30T17:32:40.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.931+0000 7f44dde51640 1 monclient: found mon.noname-c 2023-10-30T17:32:40.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.931+0000 7f44dde51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:40.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.931+0000 7f44dde51640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:40.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.931+0000 7f44dde51640 1 -- 172.21.15.170:0/3333439696 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d0009a70 con 0x7f44d8144f50 2023-10-30T17:32:40.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.933+0000 7f44dde51640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d812e500 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f44d0037040 tx=0x7f44d0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:40.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.933+0000 7f44cdffb640 10 client.?.objecter ms_handle_connect 0x7f44d8144f50 2023-10-30T17:32:40.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.933+0000 7f44cdffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:40.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44cdffb640 1 -- 172.21.15.170:0/3333439696 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d003e610 con 0x7f44d8144f50 2023-10-30T17:32:40.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:40.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:40.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44cdffb640 10 monclient: dump: 2023-10-30T17:32:40.958 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:40.959 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.959 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.959 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.959 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:40.959 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.960 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:40.960 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:40.960 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:40.960 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44cdffb640 10 monclient: _finish_auth 0 2023-10-30T17:32:40.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44cdffb640 10 monclient: _check_auth_tickets 2023-10-30T17:32:40.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:40.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44df8b4640 5 monclient: authenticate success, global_id 4322 2023-10-30T17:32:40.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44df8b4640 10 monclient: _renew_subs 2023-10-30T17:32:40.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44df8b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:40.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44df8b4640 1 -- 172.21.15.170:0/3333439696 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d813f830 con 0x7f44d8144f50 2023-10-30T17:32:40.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44df8b4640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:40.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44df8b4640 10 monclient: _renew_subs 2023-10-30T17:32:40.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44df8b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:40.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44df8b4640 1 -- 172.21.15.170:0/3333439696 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d812ec20 con 0x7f44d8144f50 2023-10-30T17:32:40.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.935+0000 7f44df8b4640 1 librados: init done 2023-10-30T17:32:40.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.936+0000 7f44df8b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:40.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.936+0000 7f44df8b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:40.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.936+0000 7f44df8b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:40.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.936+0000 7f44df8b4640 1 -- 172.21.15.170:0/3333439696 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d8141880 con 0x7f44d8144f50 2023-10-30T17:32:40.968 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.966+0000 7f63d9ba9640 10 monclient: get_monmap_and_config 2023-10-30T17:32:40.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.966+0000 7f63d9ba9640 10 monclient: build_initial_monmap 2023-10-30T17:32:40.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.966+0000 7f63d9ba9640 10 monclient: monmap: 2023-10-30T17:32:40.969 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:40.970 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.970 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:40.968708+0000 2023-10-30T17:32:40.970 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:40.968708+0000 2023-10-30T17:32:40.970 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:40.971 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.971 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:40.971 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:40.971 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:40.971 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:40.971 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:40.972 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.967+0000 7f63d9ba9640 1 Processor -- start 2023-10-30T17:32:40.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.967+0000 7f63d9ba9640 1 -- start start 2023-10-30T17:32:40.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.967+0000 7f63d9ba9640 10 monclient: init 2023-10-30T17:32:40.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:40.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-10-30T17:32:40.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d40a8720 0x7f63d413ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 10 monclient(hunting): picked mon.noname-b con 0x7f63d40a8720 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:40.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f63d413f4d0 0x7f63d41454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 10 monclient(hunting): picked mon.noname-c con 0x7f63d413f4d0 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:40.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63d40a8b00 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:40.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d40a9fd0 con 0x7f63d40a8b00 2023-10-30T17:32:40.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d40aa240 con 0x7f63d413f4d0 2023-10-30T17:32:40.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40aa520 con 0x7f63d40a8720 2023-10-30T17:32:40.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:40.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d9ba9640 10 monclient(hunting): authenticate will time out at 1343.292179s 2023-10-30T17:32:40.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d3fff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f63d40a8b00 0x7f63d4141b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:56804/0 (socket says 172.21.15.170:56804) 2023-10-30T17:32:40.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.968+0000 7f63d3fff640 1 -- 172.21.15.170:0/70783905 learned_addr learned my addr 172.21.15.170:0/70783905 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:40.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.971+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924606240 0 0) 0x7f63d40a9fd0 con 0x7f63d40a8b00 2023-10-30T17:32:40.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.971+0000 7f63d27fc640 10 monclient(hunting): _negotiate 2023-10-30T17:32:40.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.971+0000 7f63d27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:40.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.971+0000 7f63d27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:40.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.971+0000 7f63d27fc640 10 monclient(hunting): my global_id is 4359 2023-10-30T17:32:40.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.971+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63bc003520 con 0x7f63d40a8b00 2023-10-30T17:32:40.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1170222552 0 0) 0x7f63bc003520 con 0x7f63d40a8b00 2023-10-30T17:32:40.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d40a8720 msgr2=0x7f63d413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:40.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 --2- 172.21.15.170:0/70783905 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d40a8720 0x7f63d413ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 >> v2:172.21.15.177:3300/0 conn(0x7f63d413f4d0 msgr2=0x7f63d41454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:40.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 --2- 172.21.15.170:0/70783905 >> v2:172.21.15.177:3300/0 conn(0x7f63d413f4d0 0x7f63d41454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient: _finish_hunting 0 2023-10-30T17:32:40.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 monclient: found mon.noname-a-legacy 2023-10-30T17:32:40.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:40.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.170:6789/0 2023-10-30T17:32:40.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d40b1c80 con 0x7f63d40a8b00 2023-10-30T17:32:40.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient: _finish_auth 0 2023-10-30T17:32:40.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient: _check_auth_tickets 2023-10-30T17:32:40.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:40.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2465063691 0 0) 0x7f63cc0019f0 con 0x7f63d40a8b00 2023-10-30T17:32:40.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d9ba9640 5 monclient: authenticate success, global_id 4359 2023-10-30T17:32:40.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d9ba9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:40.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:40.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T17:32:40.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient: dump: 2023-10-30T17:32:40.984 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:40.985 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:40.985 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.985 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:40.985 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:40.985 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:40.986 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:40.986 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:40.986 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:40.986 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:40.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] but i'm connected to v1:172.21.15.170:6789/0, reconnecting 2023-10-30T17:32:40.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:40.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 >> v1:172.21.15.170:6789/0 conn(0x7f63d40a8b00 legacy=0x7f63d4141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:40.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:32:40.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 --2- 172.21.15.170:0/70783905 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d40a8720 0x7f63bc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient(hunting): picked mon.c con 0x7f63d40a8720 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:40.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 --2- 172.21.15.170:0/70783905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63bc0040d0 0x7f63bc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient(hunting): picked mon.a con 0x7f63bc0040d0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:40.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 --2- 172.21.15.170:0/70783905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63bc004e20 0x7f63bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:40.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient(hunting): picked mon.b con 0x7f63bc004e20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:40.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40aa240 con 0x7f63bc0040d0 2023-10-30T17:32:40.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40aa520 con 0x7f63bc004e20 2023-10-30T17:32:40.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:40.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc0055a0 con 0x7f63d40a8720 2023-10-30T17:32:40.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:40.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d27fc640 10 monclient(hunting): _finish_auth 0 2023-10-30T17:32:40.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.975+0000 7f63d9ba9640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:40.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.976+0000 7f63d2ffd640 1 --2- 172.21.15.170:0/70783905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63bc0040d0 0x7f63bc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:40.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.979+0000 7f63d2ffd640 10 monclient(hunting): get_auth_request con 0x7f63bc0040d0 auth_method 0 2023-10-30T17:32:40.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.979+0000 7f63d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:40.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.979+0000 7f63d2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:40.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.979+0000 7f63d2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:32:40.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.979+0000 7f63d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:40.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.979+0000 7f63d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:40.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.979+0000 7f63d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T17:32:40.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 10 monclient(hunting): handle_auth_done global_id 4359 payload 293 2023-10-30T17:32:40.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 1 -- 172.21.15.170:0/70783905 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d40a8720 msgr2=0x7f63bc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:40.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 1 --2- 172.21.15.170:0/70783905 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d40a8720 0x7f63bc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 1 -- 172.21.15.170:0/70783905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63bc004e20 msgr2=0x7f63bc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:40.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 1 --2- 172.21.15.170:0/70783905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63bc004e20 0x7f63bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:40.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 10 monclient: _finish_hunting 0 2023-10-30T17:32:40.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 1 monclient: found mon.a 2023-10-30T17:32:40.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:40.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:40.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 1 -- 172.21.15.170:0/70783905 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc003520 con 0x7f63bc0040d0 2023-10-30T17:32:40.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 10 monclient: _finish_auth 0 2023-10-30T17:32:40.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:40.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.981+0000 7f63d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:40.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.983+0000 7f63d2ffd640 1 --2- 172.21.15.170:0/70783905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63bc0040d0 0x7f63bc0047c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f63d413f8d0 tx=0x7f63bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e005525f7b2ae0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:40.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4032600 con 0x7f63bc0040d0 2023-10-30T17:32:40.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:40.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:40.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d27fc640 10 monclient: dump: 2023-10-30T17:32:40.999 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:40.999 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.000 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.000 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.000 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.000 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.000 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.001 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.001 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.001 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c402e050 con 0x7f63bc0040d0 2023-10-30T17:32:41.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:41.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d27fc640 1 -- 172.21.15.170:0/70783905 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c40329c0 con 0x7f63bc0040d0 2023-10-30T17:32:41.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d9ba9640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:41.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d9ba9640 4 set_mon_vals no callback set 2023-10-30T17:32:41.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:41.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d27fc640 10 monclient: dump: 2023-10-30T17:32:41.003 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.003 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.003 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.004 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.004 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.004 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.004 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.004 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.005 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.005 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d9ba9640 10 monclient: shutdown 2023-10-30T17:32:41.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d9ba9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:41.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d9ba9640 1 -- 172.21.15.170:0/70783905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63bc0040d0 msgr2=0x7f63bc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d9ba9640 1 --2- 172.21.15.170:0/70783905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63bc0040d0 0x7f63bc0047c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f63d413f8d0 tx=0x7f63bc0057f0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d3fff640 1 -- 172.21.15.170:0/70783905 reap_dead start 2023-10-30T17:32:41.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d9ba9640 1 -- 172.21.15.170:0/70783905 shutdown_connections 2023-10-30T17:32:41.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d9ba9640 1 -- 172.21.15.170:0/70783905 >> 172.21.15.170:0/70783905 conn(0x7f63d40b7300 msgr2=0x7f63d40b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:41.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.987+0000 7f63d9ba9640 1 -- 172.21.15.170:0/70783905 shutdown_connections 2023-10-30T17:32:41.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.988+0000 7f63d9ba9640 1 -- 172.21.15.170:0/70783905 wait complete. 2023-10-30T17:32:41.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.988+0000 7f63d9ba9640 10 monclient: build_initial_monmap 2023-10-30T17:32:41.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.988+0000 7f63d9ba9640 10 monclient: monmap: 2023-10-30T17:32:41.007 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:41.007 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:41.008 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:41.008 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:41.008 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:41.008 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.008 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:41.009 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:41.009 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:41.009 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 1 librados: starting msgr at 2023-10-30T17:32:41.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 1 librados: starting objecter 2023-10-30T17:32:41.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 1 Processor -- start 2023-10-30T17:32:41.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 1 -- start start 2023-10-30T17:32:41.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 1 librados: setting wanted keys 2023-10-30T17:32:41.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 1 librados: calling monclient init 2023-10-30T17:32:41.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 10 monclient: init 2023-10-30T17:32:41.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:41.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:32:41.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63bc0040d0 0x7f63d41cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 10 monclient(hunting): picked mon.noname-c con 0x7f63bc0040d0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:41.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.989+0000 7f63d9ba9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d41cc390 0x7f63d40ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 10 monclient(hunting): picked mon.noname-a con 0x7f63d41cc390 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:41.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d40ac6f0 0x7f63d41d9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 10 monclient(hunting): picked mon.noname-b con 0x7f63d40ac6f0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:41.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d41ca5e0 con 0x7f63d41cc390 2023-10-30T17:32:41.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d41cb420 con 0x7f63d40ac6f0 2023-10-30T17:32:41.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d41cb0b0 con 0x7f63bc0040d0 2023-10-30T17:32:41.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:41.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d9ba9640 10 monclient(hunting): authenticate will time out at 1343.314101s 2023-10-30T17:32:41.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d2ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d41cc390 0x7f63d40ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d2ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d41cc390 0x7f63d40ac1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:58682/0 (socket says 172.21.15.170:58682) 2023-10-30T17:32:41.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d2ffd640 1 -- 172.21.15.170:0/1691560852 learned_addr learned my addr 172.21.15.170:0/1691560852 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:41.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d2ffd640 10 monclient(hunting): get_auth_request con 0x7f63d41cc390 auth_method 0 2023-10-30T17:32:41.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:41.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:41.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.990+0000 7f63d2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:41.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.991+0000 7f63d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:41.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.991+0000 7f63d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:41.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.991+0000 7f63d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:41.016 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T17:32:41.017 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T17:32:41.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.994+0000 7f63d2ffd640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2023-10-30T17:32:41.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.994+0000 7f63d2ffd640 1 -- 172.21.15.170:0/1691560852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63bc0040d0 msgr2=0x7f63d41cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.994+0000 7f63d2ffd640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63bc0040d0 0x7f63d41cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.994+0000 7f63d2ffd640 1 -- 172.21.15.170:0/1691560852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d40ac6f0 msgr2=0x7f63d41d9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.994+0000 7f63d2ffd640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d40ac6f0 0x7f63d41d9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.994+0000 7f63d2ffd640 10 monclient: _finish_hunting 0 2023-10-30T17:32:41.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.994+0000 7f63d2ffd640 1 monclient: found mon.noname-a 2023-10-30T17:32:41.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.994+0000 7f63d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:41.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.994+0000 7f63d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:41.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.994+0000 7f63d2ffd640 1 -- 172.21.15.170:0/1691560852 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d41cb890 con 0x7f63d41cc390 2023-10-30T17:32:41.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.996+0000 7f63d2ffd640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d41cc390 0x7f63d40ac1b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f63c4035040 tx=0x7f63c40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.996+0000 7f63bbfff640 10 client.?.objecter ms_handle_connect 0x7f63d41cc390 2023-10-30T17:32:41.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:40.996+0000 7f63bbfff640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:41.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 1 -- 172.21.15.170:0/1691560852 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c403d620 con 0x7f63d41cc390 2023-10-30T17:32:41.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:41.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 10 monclient: dump: 2023-10-30T17:32:41.023 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.023 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.023 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.024 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.024 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.024 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.024 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.025 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.025 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.025 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 10 monclient: _finish_auth 0 2023-10-30T17:32:41.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 10 monclient: _check_auth_tickets 2023-10-30T17:32:41.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:41.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 1 -- 172.21.15.170:0/1691560852 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c403d780 con 0x7f63d41cc390 2023-10-30T17:32:41.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:41.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 1 -- 172.21.15.170:0/1691560852 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c403db30 con 0x7f63d41cc390 2023-10-30T17:32:41.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63d9ba9640 5 monclient: authenticate success, global_id 4362 2023-10-30T17:32:41.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63d9ba9640 10 monclient: _renew_subs 2023-10-30T17:32:41.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63d9ba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:41.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63d9ba9640 1 -- 172.21.15.170:0/1691560852 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d40acb70 con 0x7f63d41cc390 2023-10-30T17:32:41.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63d1ffb640 4 set_mon_vals no callback set 2023-10-30T17:32:41.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:41.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63bbfff640 10 monclient: dump: 2023-10-30T17:32:41.028 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.028 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.028 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.028 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.029 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.029 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.029 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.029 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.029 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.030 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63d9ba9640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:41.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63d9ba9640 10 monclient: _renew_subs 2023-10-30T17:32:41.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63d9ba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:41.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63d9ba9640 1 -- 172.21.15.170:0/1691560852 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d41da360 con 0x7f63d41cc390 2023-10-30T17:32:41.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.000+0000 7f63d9ba9640 1 librados: init done 2023-10-30T17:32:41.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.002+0000 7f63d9ba9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.002+0000 7f63d9ba9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.002+0000 7f63d9ba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:41.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.002+0000 7f63d9ba9640 1 -- 172.21.15.170:0/1691560852 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d41cb270 con 0x7f63d41cc390 2023-10-30T17:32:41.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.020+0000 7f63bbfff640 1 -- 172.21.15.170:0/1691560852 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f63c403dc90 con 0x7f63d41cc390 2023-10-30T17:32:41.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.020+0000 7f63bbfff640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f63c006c850 0x7f63c006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.021+0000 7f63d37fe640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f63c006c850 0x7f63c006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.021+0000 7f63d37fe640 10 monclient: get_auth_request con 0x7f63c006c850 auth_method 0 2023-10-30T17:32:41.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.021+0000 7f63d37fe640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f63c006c850 0x7f63c006ed10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009710 tx=0x7f63c8006d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.021+0000 7f63bbfff640 10 client.4362.objecter ms_handle_connect 0x7f63c006c850 2023-10-30T17:32:41.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.022+0000 7f63bbfff640 1 -- 172.21.15.170:0/1691560852 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f63c407a590 con 0x7f63d41cc390 2023-10-30T17:32:41.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.022+0000 7f63bbfff640 10 client.4362.objecter ms_dispatch 0x7f63d4000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:41.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.022+0000 7f63bbfff640 3 client.4362.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:41.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.022+0000 7f63bbfff640 3 client.4362.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:41.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.022+0000 7f63bbfff640 20 client.4362.objecter dump_active .. 0 homeless 2023-10-30T17:32:41.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.023+0000 7f707dffb640 1 -- 172.21.15.170:0/1232012392 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7078054130 con 0x7f7088145680 2023-10-30T17:32:41.036 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.023+0000 7f707dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.036 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.023+0000 7f707dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:41.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.131+0000 7f44551c9640 10 monclient: get_monmap_and_config 2023-10-30T17:32:41.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.131+0000 7f44551c9640 10 monclient: build_initial_monmap 2023-10-30T17:32:41.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.131+0000 7f44551c9640 10 monclient: monmap: 2023-10-30T17:32:41.135 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:41.135 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.135 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:41.133638+0000 2023-10-30T17:32:41.135 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:41.133638+0000 2023-10-30T17:32:41.136 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:41.136 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.136 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:41.136 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:41.136 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:41.137 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:41.137 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:41.137 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 1 Processor -- start 2023-10-30T17:32:41.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 1 -- start start 2023-10-30T17:32:41.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 10 monclient: init 2023-10-30T17:32:41.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:41.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-10-30T17:32:41.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f44500b7c90 0x7f44500a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 10 monclient(hunting): picked mon.noname-a con 0x7f44500b7c90 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:41.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f445013eb60 0x7f445013efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 10 monclient(hunting): picked mon.noname-c con 0x7f445013eb60 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:41.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4450144f50 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:41.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44500aa740 con 0x7f4450144f50 2023-10-30T17:32:41.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44500a9500 con 0x7f44500b7c90 2023-10-30T17:32:41.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4450145b10 con 0x7f445013eb60 2023-10-30T17:32:41.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:41.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.132+0000 7f44551c9640 10 monclient(hunting): authenticate will time out at 1343.457052s 2023-10-30T17:32:41.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.133+0000 7f444e59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f445013eb60 0x7f445013efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.134+0000 7f444e59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f445013eb60 0x7f445013efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:33276/0 (socket says 172.21.15.170:33276) 2023-10-30T17:32:41.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.134+0000 7f444e59c640 1 -- 172.21.15.170:0/4035133117 learned_addr learned my addr 172.21.15.170:0/4035133117 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:41.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.134+0000 7f444e59c640 10 monclient(hunting): get_auth_request con 0x7f445013eb60 auth_method 0 2023-10-30T17:32:41.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.134+0000 7f444e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:41.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.134+0000 7f444e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:41.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.134+0000 7f444e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:41.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.134+0000 7f444e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:41.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.134+0000 7f444e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:41.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.134+0000 7f444e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:41.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.136+0000 7f444e59c640 10 monclient(hunting): handle_auth_done global_id 4360 payload 274 2023-10-30T17:32:41.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.136+0000 7f444e59c640 1 -- 172.21.15.170:0/4035133117 >> v2:172.21.15.170:3300/0 conn(0x7f44500b7c90 msgr2=0x7f44500a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:41.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.136+0000 7f444e59c640 1 --2- 172.21.15.170:0/4035133117 >> v2:172.21.15.170:3300/0 conn(0x7f44500b7c90 0x7f44500a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.136+0000 7f444e59c640 1 -- 172.21.15.170:0/4035133117 >> v1:172.21.15.170:6789/0 conn(0x7f4450144f50 legacy=0x7f4450145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:41.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.136+0000 7f444e59c640 10 monclient: _finish_hunting 0 2023-10-30T17:32:41.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.136+0000 7f444e59c640 1 monclient: found mon.noname-c 2023-10-30T17:32:41.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.136+0000 7f444e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:41.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.136+0000 7f444e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:32:41.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.136+0000 7f444e59c640 1 -- 172.21.15.170:0/4035133117 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44500aa9b0 con 0x7f445013eb60 2023-10-30T17:32:41.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.138+0000 7f444e59c640 1 --2- 172.21.15.170:0/4035133117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 0x7f445013efb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f44500a8750 tx=0x7f444402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a63e50be71578a48 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 1 -- 172.21.15.170:0/4035133117 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444031870 con 0x7f445013eb60 2023-10-30T17:32:41.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:41.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 10 monclient: dump: 2023-10-30T17:32:41.150 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.150 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.150 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.151 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.151 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.151 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.151 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.151 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.152 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.152 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 10 monclient: _finish_auth 0 2023-10-30T17:32:41.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 10 monclient: _check_auth_tickets 2023-10-30T17:32:41.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:41.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 1 -- 172.21.15.170:0/4035133117 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44440319d0 con 0x7f445013eb60 2023-10-30T17:32:41.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f44551c9640 5 monclient: authenticate success, global_id 4360 2023-10-30T17:32:41.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f44551c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:41.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:41.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f44551c9640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:41.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f44551c9640 4 set_mon_vals no callback set 2023-10-30T17:32:41.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 1 -- 172.21.15.170:0/4035133117 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444031d60 con 0x7f445013eb60 2023-10-30T17:32:41.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:41.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.141+0000 7f444dd9b640 10 monclient: dump: 2023-10-30T17:32:41.155 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.155 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.155 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.155 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.156 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.156 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.156 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.156 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.156 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.156 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.142+0000 7f44551c9640 10 monclient: shutdown 2023-10-30T17:32:41.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.142+0000 7f44551c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:41.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.142+0000 7f44551c9640 1 -- 172.21.15.170:0/4035133117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 msgr2=0x7f445013efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.142+0000 7f44551c9640 1 --2- 172.21.15.170:0/4035133117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 0x7f445013efb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f44500a8750 tx=0x7f444402b910 comp rx=0 tx=0).stop 2023-10-30T17:32:41.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.142+0000 7f44551c9640 1 -- 172.21.15.170:0/4035133117 shutdown_connections 2023-10-30T17:32:41.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.142+0000 7f44551c9640 1 --2- 172.21.15.170:0/4035133117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 0x7f445013efb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.142+0000 7f44551c9640 1 --2- 172.21.15.170:0/4035133117 >> v2:172.21.15.170:3300/0 conn(0x7f44500b7c90 0x7f44500a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.142+0000 7f44551c9640 1 -- 172.21.15.170:0/4035133117 >> 172.21.15.170:0/4035133117 conn(0x7f4450071da0 msgr2=0x7f44500b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:41.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.142+0000 7f44551c9640 1 -- 172.21.15.170:0/4035133117 shutdown_connections 2023-10-30T17:32:41.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.142+0000 7f44551c9640 1 -- 172.21.15.170:0/4035133117 wait complete. 2023-10-30T17:32:41.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.143+0000 7f44551c9640 10 monclient: build_initial_monmap 2023-10-30T17:32:41.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.143+0000 7f44551c9640 10 monclient: monmap: 2023-10-30T17:32:41.159 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:41.159 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:41.160 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:41.160 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:41.160 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:41.160 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.160 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:41.161 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:41.161 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:41.161 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.143+0000 7f44551c9640 1 librados: starting msgr at 2023-10-30T17:32:41.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.143+0000 7f44551c9640 1 librados: starting objecter 2023-10-30T17:32:41.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.143+0000 7f44551c9640 1 Processor -- start 2023-10-30T17:32:41.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.143+0000 7f44551c9640 1 -- start start 2023-10-30T17:32:41.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.143+0000 7f44551c9640 1 librados: setting wanted keys 2023-10-30T17:32:41.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.143+0000 7f44551c9640 1 librados: calling monclient init 2023-10-30T17:32:41.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.143+0000 7f44551c9640 10 monclient: init 2023-10-30T17:32:41.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:41.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:32:41.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44500b7c90 0x7f44501d6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 10 monclient(hunting): picked mon.noname-a con 0x7f44500b7c90 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:41.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 0x7f44501d72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 10 monclient(hunting): picked mon.noname-b con 0x7f445013eb60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:41.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4450144f50 0x7f4450135500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 10 monclient(hunting): picked mon.noname-c con 0x7f4450144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:41.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44501d77f0 con 0x7f44500b7c90 2023-10-30T17:32:41.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445013c280 con 0x7f445013eb60 2023-10-30T17:32:41.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44501d6850 con 0x7f4450144f50 2023-10-30T17:32:41.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:41.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f44551c9640 10 monclient(hunting): authenticate will time out at 1343.468690s 2023-10-30T17:32:41.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.144+0000 7f444e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 0x7f44501d72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.145+0000 7f444e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 0x7f44501d72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:33290/0 (socket says 172.21.15.170:33290) 2023-10-30T17:32:41.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.145+0000 7f444e59c640 1 -- 172.21.15.170:0/2036270670 learned_addr learned my addr 172.21.15.170:0/2036270670 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:41.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.145+0000 7f444e59c640 10 monclient(hunting): get_auth_request con 0x7f445013eb60 auth_method 0 2023-10-30T17:32:41.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.145+0000 7f444e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:41.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.145+0000 7f444e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:41.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.145+0000 7f444e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:41.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.145+0000 7f444e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:41.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.145+0000 7f444e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:41.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.145+0000 7f444e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:41.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.148+0000 7f444e59c640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2023-10-30T17:32:41.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.148+0000 7f444e59c640 1 -- 172.21.15.170:0/2036270670 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4450144f50 msgr2=0x7f4450135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.148+0000 7f444e59c640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4450144f50 0x7f4450135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.148+0000 7f444e59c640 1 -- 172.21.15.170:0/2036270670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44500b7c90 msgr2=0x7f44501d6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.148+0000 7f444e59c640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44500b7c90 0x7f44501d6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.148+0000 7f444e59c640 10 monclient: _finish_hunting 0 2023-10-30T17:32:41.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.148+0000 7f444e59c640 1 monclient: found mon.noname-b 2023-10-30T17:32:41.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.148+0000 7f444e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:41.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.148+0000 7f444e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:41.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.148+0000 7f444e59c640 1 -- 172.21.15.170:0/2036270670 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44440094d0 con 0x7f445013eb60 2023-10-30T17:32:41.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.150+0000 7f444e59c640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 0x7f44501d72b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4444000c00 tx=0x7f44440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.150+0000 7f44377fe640 10 client.?.objecter ms_handle_connect 0x7f445013eb60 2023-10-30T17:32:41.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.150+0000 7f44377fe640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:41.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 1 -- 172.21.15.170:0/2036270670 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444403ad10 con 0x7f445013eb60 2023-10-30T17:32:41.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:41.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 10 monclient: dump: 2023-10-30T17:32:41.173 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.173 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.173 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.174 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.174 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.174 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.174 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.175 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.175 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.175 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 10 monclient: _finish_auth 0 2023-10-30T17:32:41.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 10 monclient: _check_auth_tickets 2023-10-30T17:32:41.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:41.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 1 -- 172.21.15.170:0/2036270670 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444403ae70 con 0x7f445013eb60 2023-10-30T17:32:41.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44551c9640 5 monclient: authenticate success, global_id 4363 2023-10-30T17:32:41.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44551c9640 10 monclient: _renew_subs 2023-10-30T17:32:41.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44551c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:41.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44551c9640 1 -- 172.21.15.170:0/2036270670 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4450140c70 con 0x7f445013eb60 2023-10-30T17:32:41.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:41.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 1 -- 172.21.15.170:0/2036270670 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444044610 con 0x7f445013eb60 2023-10-30T17:32:41.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:41.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44377fe640 10 monclient: dump: 2023-10-30T17:32:41.178 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.178 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.178 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.178 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.178 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.179 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.179 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.179 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.179 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.179 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f444d59a640 4 set_mon_vals no callback set 2023-10-30T17:32:41.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44551c9640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:41.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44551c9640 10 monclient: _renew_subs 2023-10-30T17:32:41.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44551c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:41.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44551c9640 1 -- 172.21.15.170:0/2036270670 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4450135a40 con 0x7f445013eb60 2023-10-30T17:32:41.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.153+0000 7f44551c9640 1 librados: init done 2023-10-30T17:32:41.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.155+0000 7f44551c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.155+0000 7f44551c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.155+0000 7f44551c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:41.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.155+0000 7f44551c9640 1 -- 172.21.15.170:0/2036270670 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4450067cd0 con 0x7f445013eb60 2023-10-30T17:32:41.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.186+0000 7f4718333640 10 monclient: get_monmap_and_config 2023-10-30T17:32:41.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.186+0000 7f4718333640 10 monclient: build_initial_monmap 2023-10-30T17:32:41.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.186+0000 7f4718333640 10 monclient: monmap: 2023-10-30T17:32:41.189 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:41.189 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.189 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:41.188527+0000 2023-10-30T17:32:41.190 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:41.188527+0000 2023-10-30T17:32:41.190 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:41.190 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.190 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:41.190 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:41.191 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:41.191 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:41.191 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:41.191 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 1 Processor -- start 2023-10-30T17:32:41.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 1 -- start start 2023-10-30T17:32:41.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient: init 2023-10-30T17:32:41.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:41.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T17:32:41.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f47100b7c90 0x7f47100a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient(hunting): picked mon.noname-a con 0x7f47100b7c90 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:41.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f471013eb60 0x7f471013efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient(hunting): picked mon.noname-b con 0x7f471013eb60 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:41.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4710144f50 0x7f47101453c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient(hunting): picked mon.noname-c con 0x7f4710144f50 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:41.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100a9500 con 0x7f47100b7c90 2023-10-30T17:32:41.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100b8070 con 0x7f4710144f50 2023-10-30T17:32:41.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100aa740 con 0x7f471013eb60 2023-10-30T17:32:41.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:41.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.187+0000 7f4718333640 10 monclient(hunting): authenticate will time out at 1343.512049s 2023-10-30T17:32:41.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.207+0000 7f708f56c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T17:32:41.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.207+0000 7f708f56c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T17:32:41.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.207+0000 7f708f56c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:41.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.207+0000 7f708f56c640 1 -- 172.21.15.170:0/1232012392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f70880a99e0 con 0x7f7088145680 2023-10-30T17:32:41.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.252+0000 7f44cdffb640 1 -- 172.21.15.170:0/3333439696 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d003e770 con 0x7f44d8144f50 2023-10-30T17:32:41.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.252+0000 7f44cdffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:41.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.252+0000 7f44cdffb640 1 -- 172.21.15.170:0/3333439696 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d003eae0 con 0x7f44d8144f50 2023-10-30T17:32:41.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.252+0000 7f44cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.252+0000 7f44cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:41.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.252+0000 7f44cdffb640 10 monclient: dump: 2023-10-30T17:32:41.256 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.256 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.257 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.257 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.257 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.257 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.258 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.258 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.258 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.258 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.252+0000 7f44cf7fe640 4 set_mon_vals no callback set 2023-10-30T17:32:41.259 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.253+0000 7f44cdffb640 1 -- 172.21.15.170:0/3333439696 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f44d003b050 con 0x7f44d8144f50 2023-10-30T17:32:41.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.253+0000 7f44cdffb640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f44bc06c800 0x7f44bc06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.254+0000 7f44dd650640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f44bc06c800 0x7f44bc06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.254+0000 7f44dd650640 10 monclient: get_auth_request con 0x7f44bc06c800 auth_method 0 2023-10-30T17:32:41.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.254+0000 7f44dd650640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f44bc06c800 0x7f44bc06ecc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f44d40017f0 tx=0x7f44d4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.254+0000 7f44cdffb640 10 client.4322.objecter ms_handle_connect 0x7f44bc06c800 2023-10-30T17:32:41.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.268+0000 7f44cdffb640 1 -- 172.21.15.170:0/3333439696 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f44d007b0e0 con 0x7f44d8144f50 2023-10-30T17:32:41.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.268+0000 7f44cdffb640 10 client.4322.objecter ms_dispatch 0x7f44d8000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:41.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.268+0000 7f44cdffb640 3 client.4322.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:41.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.268+0000 7f44cdffb640 3 client.4322.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:41.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.268+0000 7f44cdffb640 20 client.4322.objecter dump_active .. 0 homeless 2023-10-30T17:32:41.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.268+0000 7f44cdffb640 1 -- 172.21.15.170:0/3333439696 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f44d0077420 con 0x7f44d8144f50 2023-10-30T17:32:41.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.268+0000 7f44cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.268+0000 7f44cdffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:41.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.272+0000 7f63bbfff640 1 -- 172.21.15.170:0/1691560852 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f63c40768d0 con 0x7f63d41cc390 2023-10-30T17:32:41.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.273+0000 7f63bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.273+0000 7f63bbfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:41.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.278+0000 7f47158ce640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f471013eb60 0x7f471013efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.278+0000 7f47158ce640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f471013eb60 0x7f471013efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:35406/0 (socket says 172.21.15.170:35406) 2023-10-30T17:32:41.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.278+0000 7f47158ce640 1 -- 172.21.15.170:0/2121267475 learned_addr learned my addr 172.21.15.170:0/2121267475 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:41.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.278+0000 7f47158ce640 10 monclient(hunting): get_auth_request con 0x7f471013eb60 auth_method 0 2023-10-30T17:32:41.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.278+0000 7f47158ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:41.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.278+0000 7f47158ce640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:41.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.278+0000 7f47158ce640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:41.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.280+0000 7f47158ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:41.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.280+0000 7f47158ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:41.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.280+0000 7f47158ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:41.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.282+0000 7f47158ce640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2023-10-30T17:32:41.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.282+0000 7f47158ce640 1 -- 172.21.15.170:0/2121267475 >> v2:172.21.15.177:3300/0 conn(0x7f4710144f50 msgr2=0x7f47101453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:41.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.282+0000 7f47158ce640 1 --2- 172.21.15.170:0/2121267475 >> v2:172.21.15.177:3300/0 conn(0x7f4710144f50 0x7f47101453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.282+0000 7f47158ce640 1 -- 172.21.15.170:0/2121267475 >> v2:172.21.15.170:3300/0 conn(0x7f47100b7c90 msgr2=0x7f47100a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:41.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.282+0000 7f47158ce640 1 --2- 172.21.15.170:0/2121267475 >> v2:172.21.15.170:3300/0 conn(0x7f47100b7c90 0x7f47100a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.282+0000 7f47158ce640 10 monclient: _finish_hunting 0 2023-10-30T17:32:41.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.282+0000 7f47158ce640 1 monclient: found mon.noname-b 2023-10-30T17:32:41.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.282+0000 7f47158ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:41.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.282+0000 7f47158ce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:41.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.282+0000 7f47158ce640 1 -- 172.21.15.170:0/2121267475 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100aa9b0 con 0x7f471013eb60 2023-10-30T17:32:41.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.284+0000 7f47158ce640 1 --2- 172.21.15.170:0/2121267475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f471013eb60 0x7f471013efb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f47100b2d10 tx=0x7f470002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7725868924b302b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 1 -- 172.21.15.170:0/2121267475 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000317c0 con 0x7f471013eb60 2023-10-30T17:32:41.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:41.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 10 monclient: dump: 2023-10-30T17:32:41.291 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.292 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.292 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.292 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.292 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.292 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.293 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.293 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.293 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.293 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 10 monclient: _finish_auth 0 2023-10-30T17:32:41.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:41.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:41.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 1 -- 172.21.15.170:0/2121267475 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4700031920 con 0x7f471013eb60 2023-10-30T17:32:41.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f4718333640 5 monclient: authenticate success, global_id 4325 2023-10-30T17:32:41.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f4718333640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:41.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:41.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 1 -- 172.21.15.170:0/2121267475 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700031cb0 con 0x7f471013eb60 2023-10-30T17:32:41.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f4718333640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:41.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f4718333640 4 set_mon_vals no callback set 2023-10-30T17:32:41.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:41.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.288+0000 7f47150cd640 10 monclient: dump: 2023-10-30T17:32:41.296 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.296 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.297 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.297 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.297 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.297 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.297 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.298 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.298 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.298 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 10 monclient: shutdown 2023-10-30T17:32:41.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:41.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 1 -- 172.21.15.170:0/2121267475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f471013eb60 msgr2=0x7f471013efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 1 --2- 172.21.15.170:0/2121267475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f471013eb60 0x7f471013efb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f47100b2d10 tx=0x7f470002b860 comp rx=0 tx=0).stop 2023-10-30T17:32:41.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 1 -- 172.21.15.170:0/2121267475 shutdown_connections 2023-10-30T17:32:41.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 1 --2- 172.21.15.170:0/2121267475 >> v2:172.21.15.177:3300/0 conn(0x7f4710144f50 0x7f47101453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 1 --2- 172.21.15.170:0/2121267475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f471013eb60 0x7f471013efb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 1 --2- 172.21.15.170:0/2121267475 >> v2:172.21.15.170:3300/0 conn(0x7f47100b7c90 0x7f47100a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 1 -- 172.21.15.170:0/2121267475 >> 172.21.15.170:0/2121267475 conn(0x7f4710071da0 msgr2=0x7f47100b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:41.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 1 -- 172.21.15.170:0/2121267475 shutdown_connections 2023-10-30T17:32:41.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.289+0000 7f4718333640 1 -- 172.21.15.170:0/2121267475 wait complete. 2023-10-30T17:32:41.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.290+0000 7f4718333640 10 monclient: build_initial_monmap 2023-10-30T17:32:41.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.290+0000 7f4718333640 10 monclient: monmap: 2023-10-30T17:32:41.304 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:41.305 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:41.305 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:41.305 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:41.305 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:41.306 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.306 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:41.306 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:41.306 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:41.307 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.291+0000 7f4718333640 1 librados: starting msgr at 2023-10-30T17:32:41.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.291+0000 7f4718333640 1 librados: starting objecter 2023-10-30T17:32:41.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.291+0000 7f4718333640 1 Processor -- start 2023-10-30T17:32:41.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.291+0000 7f4718333640 1 -- start start 2023-10-30T17:32:41.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.291+0000 7f4718333640 1 librados: setting wanted keys 2023-10-30T17:32:41.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.291+0000 7f4718333640 1 librados: calling monclient init 2023-10-30T17:32:41.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.291+0000 7f4718333640 10 monclient: init 2023-10-30T17:32:41.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:41.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:32:41.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47100b7c90 0x7f4710139a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 10 monclient(hunting): picked mon.noname-a con 0x7f47100b7c90 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:41.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f471013eb60 0x7f471013a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 10 monclient(hunting): picked mon.noname-b con 0x7f471013eb60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:41.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4710144f50 0x7f47101d9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 10 monclient(hunting): picked mon.noname-c con 0x7f4710144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:41.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47101d99d0 con 0x7f47100b7c90 2023-10-30T17:32:41.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4710140980 con 0x7f471013eb60 2023-10-30T17:32:41.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471013ab10 con 0x7f4710144f50 2023-10-30T17:32:41.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:41.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f4718333640 10 monclient(hunting): authenticate will time out at 1343.616519s 2023-10-30T17:32:41.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.292+0000 7f47168d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4710144f50 0x7f47101d9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.293+0000 7f47168d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4710144f50 0x7f47101d9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:35412/0 (socket says 172.21.15.170:35412) 2023-10-30T17:32:41.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.293+0000 7f47168d0640 1 -- 172.21.15.170:0/3527034932 learned_addr learned my addr 172.21.15.170:0/3527034932 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:41.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.293+0000 7f47168d0640 10 monclient(hunting): get_auth_request con 0x7f4710144f50 auth_method 0 2023-10-30T17:32:41.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.293+0000 7f47168d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:41.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.293+0000 7f47168d0640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:41.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.293+0000 7f47168d0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:41.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.294+0000 7f47168d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:41.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.294+0000 7f47168d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:41.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.294+0000 7f47168d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:41.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.298+0000 7f47168d0640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2023-10-30T17:32:41.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.298+0000 7f47168d0640 1 -- 172.21.15.170:0/3527034932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f471013eb60 msgr2=0x7f471013a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.298+0000 7f47168d0640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f471013eb60 0x7f471013a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.298+0000 7f47168d0640 1 -- 172.21.15.170:0/3527034932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47100b7c90 msgr2=0x7f4710139a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.298+0000 7f47168d0640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47100b7c90 0x7f4710139a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.298+0000 7f47168d0640 10 monclient: _finish_hunting 0 2023-10-30T17:32:41.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.298+0000 7f47168d0640 1 monclient: found mon.noname-c 2023-10-30T17:32:41.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.298+0000 7f47168d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:41.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.298+0000 7f47168d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:41.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.298+0000 7f47168d0640 1 -- 172.21.15.170:0/3527034932 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47000094d0 con 0x7f4710144f50 2023-10-30T17:32:41.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.302+0000 7f47168d0640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4710144f50 0x7f47101d9ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f470800b970 tx=0x7f470800be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.302+0000 7f4706ffd640 10 client.?.objecter ms_handle_connect 0x7f4710144f50 2023-10-30T17:32:41.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.302+0000 7f4706ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:41.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.308+0000 7f4706ffd640 1 -- 172.21.15.170:0/3527034932 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470800cb40 con 0x7f4710144f50 2023-10-30T17:32:41.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.308+0000 7f4706ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.308+0000 7f4706ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:41.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.308+0000 7f4706ffd640 10 monclient: dump: 2023-10-30T17:32:41.319 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.319 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.319 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.319 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.320 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.320 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.320 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.320 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.320 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.320 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.308+0000 7f4706ffd640 10 monclient: _finish_auth 0 2023-10-30T17:32:41.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4706ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:32:41.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4706ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:41.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4706ffd640 1 -- 172.21.15.170:0/3527034932 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470800ccc0 con 0x7f4710144f50 2023-10-30T17:32:41.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4706ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:41.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4706ffd640 1 -- 172.21.15.170:0/3527034932 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4708014680 con 0x7f4710144f50 2023-10-30T17:32:41.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4706ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4706ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:41.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4706ffd640 10 monclient: dump: 2023-10-30T17:32:41.323 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.323 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.323 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.323 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.323 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.324 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.324 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.324 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.324 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.324 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4718333640 5 monclient: authenticate success, global_id 4328 2023-10-30T17:32:41.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f47148cc640 4 set_mon_vals no callback set 2023-10-30T17:32:41.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4718333640 10 monclient: _renew_subs 2023-10-30T17:32:41.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4718333640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:41.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4718333640 1 -- 172.21.15.170:0/3527034932 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471013c280 con 0x7f4710144f50 2023-10-30T17:32:41.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4718333640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:41.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4718333640 10 monclient: _renew_subs 2023-10-30T17:32:41.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4718333640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:41.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4718333640 1 -- 172.21.15.170:0/3527034932 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47101da610 con 0x7f4710144f50 2023-10-30T17:32:41.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.309+0000 7f4718333640 1 librados: init done 2023-10-30T17:32:41.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.310+0000 7f4718333640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.310+0000 7f4718333640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.310+0000 7f4718333640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:41.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.310+0000 7f4718333640 1 -- 172.21.15.170:0/3527034932 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4710067cd0 con 0x7f4710144f50 2023-10-30T17:32:41.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.396+0000 7f44377fe640 1 -- 172.21.15.170:0/2036270670 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4444044770 con 0x7f445013eb60 2023-10-30T17:32:41.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.396+0000 7f44377fe640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f443c06c850 0x7f443c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.396+0000 7f444ed9d640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f443c06c850 0x7f443c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.396+0000 7f444ed9d640 10 monclient: get_auth_request con 0x7f443c06c850 auth_method 0 2023-10-30T17:32:41.400 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.397+0000 7f44377fe640 10 client.4363.objecter ms_handle_connect 0x7f443c06c850 2023-10-30T17:32:41.400 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.397+0000 7f444ed9d640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f443c06c850 0x7f443c06ed10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4440009710 tx=0x7f4440006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.404 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.402+0000 7f44377fe640 1 -- 172.21.15.170:0/2036270670 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f444402d040 con 0x7f445013eb60 2023-10-30T17:32:41.404 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.402+0000 7f44377fe640 10 client.4363.objecter ms_dispatch 0x7f4450000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:41.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.402+0000 7f44377fe640 3 client.4363.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:41.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.402+0000 7f44377fe640 3 client.4363.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:41.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.402+0000 7f44377fe640 20 client.4363.objecter dump_active .. 0 homeless 2023-10-30T17:32:41.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.411+0000 7f44377fe640 1 -- 172.21.15.170:0/2036270670 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f44440776b0 con 0x7f445013eb60 2023-10-30T17:32:41.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.411+0000 7f44377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.411+0000 7f44377fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:41.418 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.416+0000 7f707dffb640 1 -- 172.21.15.170:0/1232012392 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7078057750 con 0x7f7088145680 2023-10-30T17:32:41.418 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.416+0000 7f707dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T17:32:41.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.416+0000 7f707dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:41.419 INFO:teuthology.orchestra.run.smithi170.stdout:68719476740 2023-10-30T17:32:41.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 10 librados: watch_flush enter 2023-10-30T17:32:41.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 10 librados: watch_flush exit 2023-10-30T17:32:41.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 20 client.4357.objecter shutdown clearing up homeless session... 2023-10-30T17:32:41.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 10 client.4357.objecter successfully canceled tick 2023-10-30T17:32:41.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 1 -- 172.21.15.170:0/1232012392 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f706c06c880 msgr2=0x7f706c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f706c06c880 0x7f706c06ed40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7084001840 tx=0x7f7084009040 comp rx=0 tx=0).stop 2023-10-30T17:32:41.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 10 monclient: shutdown 2023-10-30T17:32:41.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:41.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 1 -- 172.21.15.170:0/1232012392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7088145680 msgr2=0x7f70880b5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7088145680 0x7f70880b5e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f707800c3c0 tx=0x7f7078001e70 comp rx=0 tx=0).stop 2023-10-30T17:32:41.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 1 -- 172.21.15.170:0/1232012392 shutdown_connections 2023-10-30T17:32:41.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f706c06c880 0x7f706c06ed40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7088145680 0x7f70880b5e30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7088138470 0x7f70881388d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 1 --2- 172.21.15.170:0/1232012392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70880b4530 0x7f70880b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.419+0000 7f708f56c640 1 -- 172.21.15.170:0/1232012392 >> 172.21.15.170:0/1232012392 conn(0x7f7088071b90 msgr2=0x7f708813ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:41.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.420+0000 7f708f56c640 1 -- 172.21.15.170:0/1232012392 shutdown_connections 2023-10-30T17:32:41.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.420+0000 7f708f56c640 1 -- 172.21.15.170:0/1232012392 wait complete. 2023-10-30T17:32:41.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.420+0000 7f708f56c640 1 librados: shutdown 2023-10-30T17:32:41.438 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476740 for osd.3 2023-10-30T17:32:41.438 DEBUG:teuthology.parallel:result is None 2023-10-30T17:32:41.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.460+0000 7f44df8b4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T17:32:41.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.460+0000 7f44df8b4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T17:32:41.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.460+0000 7f44df8b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:41.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.460+0000 7f44df8b4640 1 -- 172.21.15.170:0/3333439696 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f44d8141880 con 0x7f44d8144f50 2023-10-30T17:32:41.479 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.477+0000 7f63d9ba9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T17:32:41.479 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.477+0000 7f63d9ba9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T17:32:41.479 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.477+0000 7f63d9ba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:41.480 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.477+0000 7f63d9ba9640 1 -- 172.21.15.170:0/1691560852 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f63d41ca5e0 con 0x7f63d41cc390 2023-10-30T17:32:41.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.528+0000 7f63bbfff640 1 -- 172.21.15.170:0/1691560852 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f63c4037020 con 0x7f63d41cc390 2023-10-30T17:32:41.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.528+0000 7f63bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T17:32:41.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.528+0000 7f63bbfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:41.532 INFO:teuthology.orchestra.run.smithi170.stdout:68719476742 2023-10-30T17:32:41.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 10 librados: watch_flush enter 2023-10-30T17:32:41.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 10 librados: watch_flush exit 2023-10-30T17:32:41.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 20 client.4362.objecter shutdown clearing up homeless session... 2023-10-30T17:32:41.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 10 client.4362.objecter successfully canceled tick 2023-10-30T17:32:41.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 -- 172.21.15.170:0/1691560852 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f63c006c850 msgr2=0x7f63c006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f63c006c850 0x7f63c006ed10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009710 tx=0x7f63c8006d10 comp rx=0 tx=0).stop 2023-10-30T17:32:41.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 10 monclient: shutdown 2023-10-30T17:32:41.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:41.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 -- 172.21.15.170:0/1691560852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d41cc390 msgr2=0x7f63d40ac1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d41cc390 0x7f63d40ac1b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f63c4035040 tx=0x7f63c40047d0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 -- 172.21.15.170:0/1691560852 shutdown_connections 2023-10-30T17:32:41.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f63c006c850 0x7f63c006ed10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d40ac6f0 0x7f63d41d9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d41cc390 0x7f63d40ac1b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 --2- 172.21.15.170:0/1691560852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63bc0040d0 0x7f63d41cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 -- 172.21.15.170:0/1691560852 >> 172.21.15.170:0/1691560852 conn(0x7f63d40b7300 msgr2=0x7f63d40b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:41.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 -- 172.21.15.170:0/1691560852 shutdown_connections 2023-10-30T17:32:41.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.531+0000 7f63d9ba9640 1 -- 172.21.15.170:0/1691560852 wait complete. 2023-10-30T17:32:41.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.532+0000 7f63d9ba9640 1 librados: shutdown 2023-10-30T17:32:41.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.1 2023-10-30T17:32:41.545 DEBUG:teuthology.parallel:result is None 2023-10-30T17:32:41.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.597+0000 7f44551c9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T17:32:41.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.597+0000 7f44551c9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T17:32:41.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.597+0000 7f44551c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:41.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.597+0000 7f44551c9640 1 -- 172.21.15.170:0/2036270670 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f44500b60d0 con 0x7f445013eb60 2023-10-30T17:32:41.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.600+0000 7f44377fe640 1 -- 172.21.15.170:0/2036270670 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4444052070 con 0x7f445013eb60 2023-10-30T17:32:41.603 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.600+0000 7f44377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-10-30T17:32:41.603 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.600+0000 7f44377fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:41.603 INFO:teuthology.orchestra.run.smithi170.stdout:60129542148 2023-10-30T17:32:41.604 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.601+0000 7f4706ffd640 1 -- 172.21.15.170:0/3527034932 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f470801d3b0 con 0x7f4710144f50 2023-10-30T17:32:41.604 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f4706ffd640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f46f406c800 0x7f46f406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 10 librados: watch_flush enter 2023-10-30T17:32:41.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 10 librados: watch_flush exit 2023-10-30T17:32:41.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 20 client.4363.objecter shutdown clearing up homeless session... 2023-10-30T17:32:41.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 10 client.4363.objecter successfully canceled tick 2023-10-30T17:32:41.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f47160cf640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f46f406c800 0x7f46f406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f47160cf640 10 monclient: get_auth_request con 0x7f46f406c800 auth_method 0 2023-10-30T17:32:41.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 1 -- 172.21.15.170:0/2036270670 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f443c06c850 msgr2=0x7f443c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f443c06c850 0x7f443c06ed10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4440009710 tx=0x7f4440006c90 comp rx=0 tx=0).stop 2023-10-30T17:32:41.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 10 monclient: shutdown 2023-10-30T17:32:41.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:41.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 1 -- 172.21.15.170:0/2036270670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 msgr2=0x7f44501d72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 0x7f44501d72b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4444000c00 tx=0x7f44440046c0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 1 -- 172.21.15.170:0/2036270670 shutdown_connections 2023-10-30T17:32:41.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f443c06c850 0x7f443c06ed10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f445013eb60 0x7f44501d72b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4450144f50 0x7f4450135500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 1 --2- 172.21.15.170:0/2036270670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44500b7c90 0x7f44501d6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.602+0000 7f44551c9640 1 -- 172.21.15.170:0/2036270670 >> 172.21.15.170:0/2036270670 conn(0x7f4450071da0 msgr2=0x7f44501453e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:41.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.603+0000 7f44551c9640 1 -- 172.21.15.170:0/2036270670 shutdown_connections 2023-10-30T17:32:41.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.603+0000 7f44551c9640 1 -- 172.21.15.170:0/2036270670 wait complete. 2023-10-30T17:32:41.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.603+0000 7f44551c9640 1 librados: shutdown 2023-10-30T17:32:41.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.603+0000 7f4706ffd640 10 client.4328.objecter ms_handle_connect 0x7f46f406c800 2023-10-30T17:32:41.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.603+0000 7f47160cf640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f46f406c800 0x7f46f406ecc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f470c009710 tx=0x7f470c006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.615 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.7 2023-10-30T17:32:41.616 DEBUG:teuthology.parallel:result is None 2023-10-30T17:32:41.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.615+0000 7f4706ffd640 1 -- 172.21.15.170:0/3527034932 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f470800ce30 con 0x7f4710144f50 2023-10-30T17:32:41.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.615+0000 7f4706ffd640 10 client.4328.objecter ms_dispatch 0x7f4710000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:41.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.615+0000 7f4706ffd640 3 client.4328.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:41.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.615+0000 7f4706ffd640 3 client.4328.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:41.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.615+0000 7f4706ffd640 20 client.4328.objecter dump_active .. 0 homeless 2023-10-30T17:32:41.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.615+0000 7f4706ffd640 1 -- 172.21.15.170:0/3527034932 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4708054ad0 con 0x7f4710144f50 2023-10-30T17:32:41.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.616+0000 7f4706ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.616+0000 7f4706ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:41.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.686+0000 7f44cdffb640 1 -- 172.21.15.170:0/3333439696 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f44d8141880 con 0x7f44d8144f50 2023-10-30T17:32:41.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.686+0000 7f44cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T17:32:41.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.686+0000 7f44cdffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:41.690 INFO:teuthology.orchestra.run.smithi170.stdout:73014444036 2023-10-30T17:32:41.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 10 librados: watch_flush enter 2023-10-30T17:32:41.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 10 librados: watch_flush exit 2023-10-30T17:32:41.692 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 20 client.4322.objecter shutdown clearing up homeless session... 2023-10-30T17:32:41.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 10 client.4322.objecter successfully canceled tick 2023-10-30T17:32:41.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 -- 172.21.15.170:0/3333439696 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f44bc06c800 msgr2=0x7f44bc06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f44bc06c800 0x7f44bc06ecc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f44d40017f0 tx=0x7f44d4009040 comp rx=0 tx=0).stop 2023-10-30T17:32:41.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 10 monclient: shutdown 2023-10-30T17:32:41.693 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:41.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 -- 172.21.15.170:0/3333439696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 msgr2=0x7f44d812e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d812e500 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f44d0037040 tx=0x7f44d0004a40 comp rx=0 tx=0).stop 2023-10-30T17:32:41.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 -- 172.21.15.170:0/3333439696 shutdown_connections 2023-10-30T17:32:41.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f44bc06c800 0x7f44bc06ecc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44d813eb60 0x7f44d8133c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44d8144f50 0x7f44d812e500 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 --2- 172.21.15.170:0/3333439696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44d80b7c90 0x7f44d813fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 -- 172.21.15.170:0/3333439696 >> 172.21.15.170:0/3333439696 conn(0x7f44d8071da0 msgr2=0x7f44d80b6700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:41.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 -- 172.21.15.170:0/3333439696 shutdown_connections 2023-10-30T17:32:41.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 -- 172.21.15.170:0/3333439696 wait complete. 2023-10-30T17:32:41.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.689+0000 7f44df8b4640 1 librados: shutdown 2023-10-30T17:32:41.706 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444036 for osd.0 2023-10-30T17:32:41.707 DEBUG:teuthology.parallel:result is None 2023-10-30T17:32:41.810 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.808+0000 7f4718333640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T17:32:41.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.808+0000 7f4718333640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T17:32:41.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.808+0000 7f4718333640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:41.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.808+0000 7f4718333640 1 -- 172.21.15.170:0/3527034932 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f471013c470 con 0x7f4710144f50 2023-10-30T17:32:41.813 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.811+0000 7f4706ffd640 1 -- 172.21.15.170:0/3527034932 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4708019070 con 0x7f4710144f50 2023-10-30T17:32:41.813 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.811+0000 7f4706ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T17:32:41.814 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.811+0000 7f4706ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:41.814 INFO:teuthology.orchestra.run.smithi170.stdout:64424509444 2023-10-30T17:32:41.816 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 10 librados: watch_flush enter 2023-10-30T17:32:41.816 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 10 librados: watch_flush exit 2023-10-30T17:32:41.816 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 20 client.4328.objecter shutdown clearing up homeless session... 2023-10-30T17:32:41.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 10 client.4328.objecter successfully canceled tick 2023-10-30T17:32:41.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 -- 172.21.15.170:0/3527034932 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f46f406c800 msgr2=0x7f46f406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f46f406c800 0x7f46f406ecc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f470c009710 tx=0x7f470c006c90 comp rx=0 tx=0).stop 2023-10-30T17:32:41.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 10 monclient: shutdown 2023-10-30T17:32:41.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:41.818 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 -- 172.21.15.170:0/3527034932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4710144f50 msgr2=0x7f47101d9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.818 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4710144f50 0x7f47101d9ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f470800b970 tx=0x7f470800be30 comp rx=0 tx=0).stop 2023-10-30T17:32:41.818 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 -- 172.21.15.170:0/3527034932 shutdown_connections 2023-10-30T17:32:41.818 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f46f406c800 0x7f46f406ecc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.819 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f471013eb60 0x7f471013a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.819 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4710144f50 0x7f47101d9ef0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.819 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 --2- 172.21.15.170:0/3527034932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47100b7c90 0x7f4710139a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.819 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 -- 172.21.15.170:0/3527034932 >> 172.21.15.170:0/3527034932 conn(0x7f4710071da0 msgr2=0x7f471013d900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:41.819 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 -- 172.21.15.170:0/3527034932 shutdown_connections 2023-10-30T17:32:41.820 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 -- 172.21.15.170:0/3527034932 wait complete. 2023-10-30T17:32:41.820 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.813+0000 7f4718333640 1 librados: shutdown 2023-10-30T17:32:41.827 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.2 2023-10-30T17:32:41.828 DEBUG:teuthology.parallel:result is None 2023-10-30T17:32:41.828 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T17:32:41.828 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T17:32:41.930 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.927+0000 7f2fb75e9640 10 monclient: get_monmap_and_config 2023-10-30T17:32:41.930 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.927+0000 7f2fb75e9640 10 monclient: build_initial_monmap 2023-10-30T17:32:41.930 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.927+0000 7f2fb75e9640 10 monclient: monmap: 2023-10-30T17:32:41.930 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:41.931 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.931 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:41.929697+0000 2023-10-30T17:32:41.931 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:41.929697+0000 2023-10-30T17:32:41.931 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:41.931 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.932 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:41.932 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:41.932 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:41.932 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:41.932 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:41.933 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.928+0000 7f2fb75e9640 1 Processor -- start 2023-10-30T17:32:41.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.928+0000 7f2fb75e9640 1 -- start start 2023-10-30T17:32:41.936 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.928+0000 7f2fb75e9640 10 monclient: init 2023-10-30T17:32:41.936 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:41.936 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T17:32:41.936 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2fb00a84b0 0x7f2fb013f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.936 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb00a84b0 addr v2:172.21.15.170:3300/0 2023-10-30T17:32:41.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fb00a8890 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:41.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2fb00a9250 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:41.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb00aa9d0 con 0x7f2fb00a8890 2023-10-30T17:32:41.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb00b1570 con 0x7f2fb00a9250 2023-10-30T17:32:41.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb00a9670 con 0x7f2fb00a84b0 2023-10-30T17:32:41.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:41.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb75e9640 10 monclient(hunting): authenticate will time out at 1344.253241s 2023-10-30T17:32:41.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.929+0000 7f2fb5385640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2fb00a84b0 0x7f2fb013f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.930+0000 7f2fb5b86640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f2fb00a9250 0x7f2fb0142420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:46926/0 (socket says 172.21.15.170:46926) 2023-10-30T17:32:41.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.930+0000 7f2fb5b86640 1 -- 172.21.15.170:0/3226200843 learned_addr learned my addr 172.21.15.170:0/3226200843 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:41.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.930+0000 7f2fb5385640 10 monclient(hunting): get_auth_request con 0x7f2fb00a84b0 auth_method 0 2023-10-30T17:32:41.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.930+0000 7f2fb5385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:41.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.930+0000 7f2fb5385640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:41.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.930+0000 7f2fb5385640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:41.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.931+0000 7f2fb5385640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:41.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.931+0000 7f2fb5385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:41.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.931+0000 7f2fb5385640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:41.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.932+0000 7f2fa7fff640 1 -- 172.21.15.170:0/3226200843 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652170420 0 0) 0x7f2fb00b1570 con 0x7f2fb00a9250 2023-10-30T17:32:41.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.932+0000 7f2fa7fff640 10 monclient(hunting): _negotiate 2023-10-30T17:32:41.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.932+0000 7f2fa7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:41.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.932+0000 7f2fa7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:41.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.932+0000 7f2fa7fff640 10 monclient(hunting): my global_id is 4366 2023-10-30T17:32:41.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.932+0000 7f2fa7fff640 1 -- 172.21.15.170:0/3226200843 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c003520 con 0x7f2fb00a9250 2023-10-30T17:32:41.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.933+0000 7f2fb5385640 10 monclient(hunting): handle_auth_done global_id 4365 payload 274 2023-10-30T17:32:41.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.933+0000 7f2fb5385640 1 -- 172.21.15.170:0/3226200843 >> v1:172.21.15.177:6789/0 conn(0x7f2fb00a9250 legacy=0x7f2fb0142420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.933+0000 7f2fb5385640 1 -- 172.21.15.170:0/3226200843 >> v1:172.21.15.170:6789/0 conn(0x7f2fb00a8890 legacy=0x7f2fb0141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.933+0000 7f2fb5385640 10 monclient: _finish_hunting 0 2023-10-30T17:32:41.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.933+0000 7f2fb5385640 1 monclient: found mon.noname-a 2023-10-30T17:32:41.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.933+0000 7f2fb5385640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:41.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.933+0000 7f2fb5385640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:41.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.933+0000 7f2fb5385640 1 -- 172.21.15.170:0/3226200843 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb00ae150 con 0x7f2fb00a84b0 2023-10-30T17:32:41.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.937+0000 7f2fb5385640 1 --2- 172.21.15.170:0/3226200843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a84b0 0x7f2fb013f170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0097a0 tx=0x7f2fac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88ee652d92d856f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 1 -- 172.21.15.170:0/3226200843 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac02e070 con 0x7f2fb00a84b0 2023-10-30T17:32:41.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:41.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 10 monclient: dump: 2023-10-30T17:32:41.946 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.946 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.946 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.947 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.947 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.947 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.947 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.947 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.948 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.948 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 10 monclient: _finish_auth 0 2023-10-30T17:32:41.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 10 monclient: _check_auth_tickets 2023-10-30T17:32:41.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:41.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 1 -- 172.21.15.170:0/3226200843 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac0315a0 con 0x7f2fb00a84b0 2023-10-30T17:32:41.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:41.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 1 -- 172.21.15.170:0/3226200843 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac031930 con 0x7f2fb00a84b0 2023-10-30T17:32:41.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:41.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fa7fff640 10 monclient: dump: 2023-10-30T17:32:41.950 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.950 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.950 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.950 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.950 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.951 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.951 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.951 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.951 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.951 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fb75e9640 5 monclient: authenticate success, global_id 4365 2023-10-30T17:32:41.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fb75e9640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:41.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.943+0000 7f2fb75e9640 4 set_mon_vals no callback set 2023-10-30T17:32:41.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.944+0000 7f2fb75e9640 10 monclient: shutdown 2023-10-30T17:32:41.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.944+0000 7f2fb75e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:41.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.944+0000 7f2fb75e9640 1 -- 172.21.15.170:0/3226200843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a84b0 msgr2=0x7f2fb013f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.944+0000 7f2fb75e9640 1 --2- 172.21.15.170:0/3226200843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a84b0 0x7f2fb013f170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0097a0 tx=0x7f2fac02b840 comp rx=0 tx=0).stop 2023-10-30T17:32:41.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.944+0000 7f2fb75e9640 1 -- 172.21.15.170:0/3226200843 shutdown_connections 2023-10-30T17:32:41.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.944+0000 7f2fb75e9640 1 --2- 172.21.15.170:0/3226200843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a84b0 0x7f2fb013f170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.944+0000 7f2fb75e9640 1 -- 172.21.15.170:0/3226200843 >> 172.21.15.170:0/3226200843 conn(0x7f2fb00b7330 msgr2=0x7f2fb00b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:41.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.944+0000 7f2fb75e9640 1 -- 172.21.15.170:0/3226200843 shutdown_connections 2023-10-30T17:32:41.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.944+0000 7f2fb75e9640 1 -- 172.21.15.170:0/3226200843 wait complete. 2023-10-30T17:32:41.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 10 monclient: build_initial_monmap 2023-10-30T17:32:41.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 10 monclient: monmap: 2023-10-30T17:32:41.958 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:41.958 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:41.958 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:41.959 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:41.959 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:41.959 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.959 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:41.959 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:41.960 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:41.960 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 1 librados: starting msgr at 2023-10-30T17:32:41.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 1 librados: starting objecter 2023-10-30T17:32:41.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 1 Processor -- start 2023-10-30T17:32:41.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 1 -- start start 2023-10-30T17:32:41.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 1 librados: setting wanted keys 2023-10-30T17:32:41.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 1 librados: calling monclient init 2023-10-30T17:32:41.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 10 monclient: init 2023-10-30T17:32:41.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:41.961 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.945+0000 7f2fb75e9640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:32:41.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fb00a84b0 0x7f2fb00ab110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb00a84b0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:41.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a9250 0x7f2fb01d66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb00a9250 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:41.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fb0142180 0x7f2fb00ad140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb0142180 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:41.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00adc40 con 0x7f2fb00a9250 2023-10-30T17:32:41.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb01d6be0 con 0x7f2fb0142180 2023-10-30T17:32:41.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:41.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0141560 con 0x7f2fb00a84b0 2023-10-30T17:32:41.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:41.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb75e9640 10 monclient(hunting): authenticate will time out at 1344.270113s 2023-10-30T17:32:41.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb4b84640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a9250 0x7f2fb01d66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb5b86640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fb0142180 0x7f2fb00ad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb4b84640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a9250 0x7f2fb01d66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:58744/0 (socket says 172.21.15.170:58744) 2023-10-30T17:32:41.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb4b84640 1 -- 172.21.15.170:0/2425033593 learned_addr learned my addr 172.21.15.170:0/2425033593 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:41.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb4b84640 10 monclient(hunting): get_auth_request con 0x7f2fb00a9250 auth_method 0 2023-10-30T17:32:41.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb4b84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:41.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb4b84640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:41.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.946+0000 7f2fb4b84640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:41.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5385640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fb00a84b0 0x7f2fb00ab110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5385640 10 monclient(hunting): get_auth_request con 0x7f2fb00a84b0 auth_method 0 2023-10-30T17:32:41.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:41.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5385640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:41.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5385640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:41.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb4b84640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:41.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb4b84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:41.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb4b84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:41.968 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5b86640 10 monclient(hunting): get_auth_request con 0x7f2fb0142180 auth_method 0 2023-10-30T17:32:41.968 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5b86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:41.968 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5b86640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:41.968 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5b86640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:41.968 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5385640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:41.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:41.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.947+0000 7f2fb5385640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:41.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.949+0000 7f2fb5b86640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:41.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.949+0000 7f2fb5b86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:41.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.949+0000 7f2fb5b86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:41.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.950+0000 7f2fb4b84640 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2023-10-30T17:32:41.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.950+0000 7f2fb4b84640 1 -- 172.21.15.170:0/2425033593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fb00a84b0 msgr2=0x7f2fb00ab110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.950+0000 7f2fb4b84640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fb00a84b0 0x7f2fb00ab110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.950+0000 7f2fb4b84640 1 -- 172.21.15.170:0/2425033593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fb0142180 msgr2=0x7f2fb00ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:41.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.950+0000 7f2fb4b84640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fb0142180 0x7f2fb00ad140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:41.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.950+0000 7f2fb4b84640 10 monclient: _finish_hunting 0 2023-10-30T17:32:41.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.950+0000 7f2fb4b84640 1 monclient: found mon.noname-a 2023-10-30T17:32:41.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.950+0000 7f2fb4b84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:41.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.950+0000 7f2fb4b84640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:32:41.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.950+0000 7f2fb4b84640 1 -- 172.21.15.170:0/2425033593 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac009400 con 0x7f2fb00a9250 2023-10-30T17:32:41.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.952+0000 7f2fb4b84640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a9250 0x7f2fb01d66a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c450 tx=0x7f2fa000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.952+0000 7f2fa5ffb640 10 client.?.objecter ms_handle_connect 0x7f2fb00a9250 2023-10-30T17:32:41.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.952+0000 7f2fa5ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:41.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fa5ffb640 1 -- 172.21.15.170:0/2425033593 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa00052d0 con 0x7f2fb00a9250 2023-10-30T17:32:41.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fa5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fa5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:32:41.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fa5ffb640 10 monclient: dump: 2023-10-30T17:32:41.977 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.977 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.977 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.977 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.978 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.978 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.978 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.978 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.978 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.979 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fa5ffb640 10 monclient: _finish_auth 0 2023-10-30T17:32:41.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fa5ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:32:41.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:41.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fb75e9640 5 monclient: authenticate success, global_id 4371 2023-10-30T17:32:41.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fb75e9640 10 monclient: _renew_subs 2023-10-30T17:32:41.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fb75e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:41.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.954+0000 7f2fb75e9640 1 -- 172.21.15.170:0/2425033593 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb00aadf0 con 0x7f2fb00a9250 2023-10-30T17:32:41.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.955+0000 7f2fb75e9640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:41.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.955+0000 7f2fb75e9640 10 monclient: _renew_subs 2023-10-30T17:32:41.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.955+0000 7f2fb75e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:41.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.955+0000 7f2fb75e9640 1 -- 172.21.15.170:0/2425033593 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb00ad680 con 0x7f2fb00a9250 2023-10-30T17:32:41.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.955+0000 7f2fb75e9640 1 librados: init done 2023-10-30T17:32:41.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fb75e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fb75e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:41.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fb75e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:32:41.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fb75e9640 1 -- 172.21.15.170:0/2425033593 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb004e3c0 con 0x7f2fb00a9250 2023-10-30T17:32:41.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fa5ffb640 1 -- 172.21.15.170:0/2425033593 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa00011e0 con 0x7f2fb00a9250 2023-10-30T17:32:41.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fa5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:41.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fa5ffb640 1 -- 172.21.15.170:0/2425033593 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0007030 con 0x7f2fb00a9250 2023-10-30T17:32:41.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fa5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:41.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fa5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:32:41.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fa5ffb640 10 monclient: dump: 2023-10-30T17:32:41.983 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:41.983 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:41.983 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.984 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:41.984 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:41.984 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:41.984 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:41.984 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:41.984 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:41.985 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:41.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.956+0000 7f2fa77fe640 4 set_mon_vals no callback set 2023-10-30T17:32:41.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.975+0000 7f2fa5ffb640 1 -- 172.21.15.170:0/2425033593 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2fa0007190 con 0x7f2fb00a9250 2023-10-30T17:32:41.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.975+0000 7f2fa5ffb640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2f9406c800 0x7f2f9406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:41.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.976+0000 7f2fb5385640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2f9406c800 0x7f2f9406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:41.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.976+0000 7f2fb5385640 10 monclient: get_auth_request con 0x7f2f9406c800 auth_method 0 2023-10-30T17:32:41.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.976+0000 7f2fb5385640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2f9406c800 0x7f2f9406ecc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fac036000 tx=0x7f2fac03d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:41.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:41.976+0000 7f2fa5ffb640 10 client.4371.objecter ms_handle_connect 0x7f2f9406c800 2023-10-30T17:32:42.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.209+0000 7f2fa5ffb640 1 -- 172.21.15.170:0/2425033593 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2fa00589a0 con 0x7f2fb00a9250 2023-10-30T17:32:42.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.209+0000 7f2fa5ffb640 10 client.4371.objecter ms_dispatch 0x7f2fb0000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:42.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.209+0000 7f2fa5ffb640 3 client.4371.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:42.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.209+0000 7f2fa5ffb640 3 client.4371.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:42.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.209+0000 7f2fa5ffb640 20 client.4371.objecter dump_active .. 0 homeless 2023-10-30T17:32:42.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.209+0000 7f2fa5ffb640 1 -- 172.21.15.170:0/2425033593 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2fa0058770 con 0x7f2fb00a9250 2023-10-30T17:32:42.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.209+0000 7f2fa5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:42.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.209+0000 7f2fa5ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:42.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.372+0000 7f2fb75e9640 1 -- 172.21.15.170:0/2425033593 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2fb014d550 con 0x7f2f9406c800 2023-10-30T17:32:42.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.375+0000 7f2fa5ffb640 1 -- 172.21.15.170:0/2425033593 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47954 (secure 0 0 0) 0x7f2fac031ad0 con 0x7f2f9406c800 2023-10-30T17:32:42.378 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:32:42.379 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-10-30T17:32:42.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 10 librados: watch_flush enter 2023-10-30T17:32:42.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 10 librados: watch_flush exit 2023-10-30T17:32:42.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 20 client.4371.objecter shutdown clearing up homeless session... 2023-10-30T17:32:42.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 10 client.4371.objecter successfully canceled tick 2023-10-30T17:32:42.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 -- 172.21.15.170:0/2425033593 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2f9406c800 msgr2=0x7f2f9406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:42.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2f9406c800 0x7f2f9406ecc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fac036000 tx=0x7f2fac03d040 comp rx=0 tx=0).stop 2023-10-30T17:32:42.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 10 monclient: shutdown 2023-10-30T17:32:42.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:42.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 -- 172.21.15.170:0/2425033593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a9250 msgr2=0x7f2fb01d66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:42.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a9250 0x7f2fb01d66a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c450 tx=0x7f2fa000c910 comp rx=0 tx=0).stop 2023-10-30T17:32:42.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 -- 172.21.15.170:0/2425033593 shutdown_connections 2023-10-30T17:32:42.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2f9406c800 0x7f2f9406ecc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fb0142180 0x7f2fb00ad140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb00a9250 0x7f2fb01d66a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 --2- 172.21.15.170:0/2425033593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fb00a84b0 0x7f2fb00ab110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 -- 172.21.15.170:0/2425033593 >> 172.21.15.170:0/2425033593 conn(0x7f2fb00b7330 msgr2=0x7f2fb00b64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:42.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 -- 172.21.15.170:0/2425033593 shutdown_connections 2023-10-30T17:32:42.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 -- 172.21.15.170:0/2425033593 wait complete. 2023-10-30T17:32:42.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.376+0000 7f2fb75e9640 1 librados: shutdown 2023-10-30T17:32:42.389 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-30T17:32:40.981348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219136,"kb_used_data":2576,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755584,"statfs":{"total":773094113280,"available":772869718016,"internally_reserved":0,"allocated":2637824,"data_stored":1440848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":59,"apply_latency_ms":59,"commit_latency_ns":59000000,"apply_latency_ns":59000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.032460"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.769859+0000","last_change":"2023-10-30T17:32:35.372482+0000","last_active":"2023-10-30T17:32:36.769859+0000","last_peered":"2023-10-30T17:32:36.769859+0000","last_clean":"2023-10-30T17:32:36.769859+0000","last_became_active":"2023-10-30T17:32:35.367487+0000","last_became_peered":"2023-10-30T17:32:35.367487+0000","last_unstale":"2023-10-30T17:32:36.769859+0000","last_undegraded":"2023-10-30T17:32:36.769859+0000","last_fullsized":"2023-10-30T17:32:36.769859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:45:03.846674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.740287+0000","last_change":"2023-10-30T17:32:34.589393+0000","last_active":"2023-10-30T17:32:36.740287+0000","last_peered":"2023-10-30T17:32:36.740287+0000","last_clean":"2023-10-30T17:32:36.740287+0000","last_became_active":"2023-10-30T17:32:34.557021+0000","last_became_peered":"2023-10-30T17:32:34.557021+0000","last_unstale":"2023-10-30T17:32:36.740287+0000","last_undegraded":"2023-10-30T17:32:36.740287+0000","last_fullsized":"2023-10-30T17:32:36.740287+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:05.352859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.761366+0000","last_change":"2023-10-30T17:32:35.362652+0000","last_active":"2023-10-30T17:32:36.761366+0000","last_peered":"2023-10-30T17:32:36.761366+0000","last_clean":"2023-10-30T17:32:36.761366+0000","last_became_active":"2023-10-30T17:32:35.357489+0000","last_became_peered":"2023-10-30T17:32:35.357489+0000","last_unstale":"2023-10-30T17:32:36.761366+0000","last_undegraded":"2023-10-30T17:32:36.761366+0000","last_fullsized":"2023-10-30T17:32:36.761366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:21:36.073054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.743877+0000","last_change":"2023-10-30T17:32:34.658442+0000","last_active":"2023-10-30T17:32:36.743877+0000","last_peered":"2023-10-30T17:32:36.743877+0000","last_clean":"2023-10-30T17:32:36.743877+0000","last_became_active":"2023-10-30T17:32:34.631494+0000","last_became_peered":"2023-10-30T17:32:34.631494+0000","last_unstale":"2023-10-30T17:32:36.743877+0000","last_undegraded":"2023-10-30T17:32:36.743877+0000","last_fullsized":"2023-10-30T17:32:36.743877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:36:36.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.753615+0000","last_change":"2023-10-30T17:32:35.388556+0000","last_active":"2023-10-30T17:32:36.753615+0000","last_peered":"2023-10-30T17:32:36.753615+0000","last_clean":"2023-10-30T17:32:36.753615+0000","last_became_active":"2023-10-30T17:32:35.383414+0000","last_became_peered":"2023-10-30T17:32:35.383414+0000","last_unstale":"2023-10-30T17:32:36.753615+0000","last_undegraded":"2023-10-30T17:32:36.753615+0000","last_fullsized":"2023-10-30T17:32:36.753615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:46.189212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.612306+0000","last_change":"2023-10-30T17:32:34.318569+0000","last_active":"2023-10-30T17:32:36.612306+0000","last_peered":"2023-10-30T17:32:36.612306+0000","last_clean":"2023-10-30T17:32:36.612306+0000","last_became_active":"2023-10-30T17:32:34.287031+0000","last_became_peered":"2023-10-30T17:32:34.287031+0000","last_unstale":"2023-10-30T17:32:36.612306+0000","last_undegraded":"2023-10-30T17:32:36.612306+0000","last_fullsized":"2023-10-30T17:32:36.612306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:55:59.080654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.748690+0000","last_change":"2023-10-30T17:32:35.307241+0000","last_active":"2023-10-30T17:32:36.748690+0000","last_peered":"2023-10-30T17:32:36.748690+0000","last_clean":"2023-10-30T17:32:36.748690+0000","last_became_active":"2023-10-30T17:32:35.277163+0000","last_became_peered":"2023-10-30T17:32:35.277163+0000","last_unstale":"2023-10-30T17:32:36.748690+0000","last_undegraded":"2023-10-30T17:32:36.748690+0000","last_fullsized":"2023-10-30T17:32:36.748690+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:02.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.758421+0000","last_change":"2023-10-30T17:32:35.628758+0000","last_active":"2023-10-30T17:32:36.758421+0000","last_peered":"2023-10-30T17:32:36.758421+0000","last_clean":"2023-10-30T17:32:36.758421+0000","last_became_active":"2023-10-30T17:32:35.623395+0000","last_became_peered":"2023-10-30T17:32:35.623395+0000","last_unstale":"2023-10-30T17:32:36.758421+0000","last_undegraded":"2023-10-30T17:32:36.758421+0000","last_fullsized":"2023-10-30T17:32:36.758421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:47:08.984576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":118,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:38.071164+0000","last_change":"2023-10-30T17:32:28.719757+0000","last_active":"2023-10-30T17:32:38.071164+0000","last_peered":"2023-10-30T17:32:38.071164+0000","last_clean":"2023-10-30T17:32:38.071164+0000","last_became_active":"2023-10-30T17:32:28.686958+0000","last_became_peered":"2023-10-30T17:32:28.686958+0000","last_unstale":"2023-10-30T17:32:38.071164+0000","last_undegraded":"2023-10-30T17:32:38.071164+0000","last_fullsized":"2023-10-30T17:32:38.071164+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:25.778908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:25.778908+0000","last_clean_scrub_stamp":"2023-10-30T17:32:25.778908+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:41:21.329138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8350000000000009}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.9}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3529999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4410000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.940999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.352}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.052}]}]},{"osd":3,"up_from":16,"seq":68719476740,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2690000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6950000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.6709999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.030999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.509}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.206}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.890000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":630784,"data_stored":478486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8220000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.613}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9429999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.236000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.908000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.757000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.033999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.702}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":48.518000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.151}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.707000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.319000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.138}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.446999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.834}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2309999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.103999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.907}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.866}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.91}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.655000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.198}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.045}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3579999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.68}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.725999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.093}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.552}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.622}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.945}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.460000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.099}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.335999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.287000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.542999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":630784,"data_stored":478486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0830000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8579999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.711}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.192}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.831}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.35}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.311999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T17:32:42.391 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T17:32:42.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.491+0000 7f8c7d24b640 10 monclient: get_monmap_and_config 2023-10-30T17:32:42.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.491+0000 7f8c7d24b640 10 monclient: build_initial_monmap 2023-10-30T17:32:42.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.491+0000 7f8c7d24b640 10 monclient: monmap: 2023-10-30T17:32:42.494 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:42.494 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:42.494 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:42.492977+0000 2023-10-30T17:32:42.494 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:42.492977+0000 2023-10-30T17:32:42.494 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:42.495 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:42.495 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:42.495 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:42.495 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:42.495 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:42.495 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:42.496 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:42.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.491+0000 7f8c7d24b640 1 Processor -- start 2023-10-30T17:32:42.498 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.491+0000 7f8c7d24b640 1 -- start start 2023-10-30T17:32:42.498 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.491+0000 7f8c7d24b640 10 monclient: init 2023-10-30T17:32:42.498 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:42.498 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-10-30T17:32:42.498 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8c780a8d80 0x7f8c780b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:42.499 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 10 monclient(hunting): picked mon.noname-c con 0x7f8c780a8d80 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:42.499 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c780a9160 0x7f8c781456a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:42.499 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 10 monclient(hunting): picked mon.noname-b con 0x7f8c780a9160 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:42.499 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c78145be0 addr v1:172.21.15.170:6789/0 2023-10-30T17:32:42.499 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c78144f30 con 0x7f8c78145be0 2023-10-30T17:32:42.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:42.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c780aa660 con 0x7f8c780a8d80 2023-10-30T17:32:42.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:42.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c78145180 con 0x7f8c780a9160 2023-10-30T17:32:42.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:42.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7d24b640 10 monclient(hunting): authenticate will time out at 1344.816390s 2023-10-30T17:32:42.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.492+0000 7f8c7759e640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f8c78145be0 0x7f8c7813ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:56836/0 (socket says 172.21.15.170:56836) 2023-10-30T17:32:42.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.493+0000 7f8c7759e640 1 -- 172.21.15.170:0/1115016669 learned_addr learned my addr 172.21.15.170:0/1115016669 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:32:42.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.495+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216555789 0 0) 0x7f8c78144f30 con 0x7f8c78145be0 2023-10-30T17:32:42.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.495+0000 7f8c75d9b640 10 monclient(hunting): _negotiate 2023-10-30T17:32:42.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.495+0000 7f8c75d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:42.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.495+0000 7f8c75d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:42.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.495+0000 7f8c75d9b640 10 monclient(hunting): my global_id is 4374 2023-10-30T17:32:42.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.495+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c60003520 con 0x7f8c78145be0 2023-10-30T17:32:42.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.497+0000 7f8c76d9d640 1 --2- 172.21.15.170:0/1115016669 >> v2:172.21.15.177:3300/0 conn(0x7f8c780a8d80 0x7f8c780b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:42.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.498+0000 7f8c7659c640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c780a9160 0x7f8c781456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:42.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.498+0000 7f8c76d9d640 10 monclient(hunting): get_auth_request con 0x7f8c780a8d80 auth_method 0 2023-10-30T17:32:42.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.498+0000 7f8c76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:42.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.498+0000 7f8c76d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:42.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.498+0000 7f8c76d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:42.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.499+0000 7f8c76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:42.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.499+0000 7f8c76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:42.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.499+0000 7f8c76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:42.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1682848957 0 0) 0x7f8c60003520 con 0x7f8c78145be0 2023-10-30T17:32:42.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c780a9160 msgr2=0x7f8c781456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:42.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c780a9160 0x7f8c781456a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 >> v2:172.21.15.177:3300/0 conn(0x7f8c780a8d80 msgr2=0x7f8c780b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:42.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 --2- 172.21.15.170:0/1115016669 >> v2:172.21.15.177:3300/0 conn(0x7f8c780a8d80 0x7f8c780b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient: _finish_hunting 0 2023-10-30T17:32:42.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 monclient: found mon.noname-a-legacy 2023-10-30T17:32:42.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:42.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.170:6789/0 2023-10-30T17:32:42.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7813f400 con 0x7f8c78145be0 2023-10-30T17:32:42.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient: _finish_auth 0 2023-10-30T17:32:42.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient: _check_auth_tickets 2023-10-30T17:32:42.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:42.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2465063691 0 0) 0x7f8c680019f0 con 0x7f8c78145be0 2023-10-30T17:32:42.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:42.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T17:32:42.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient: dump: 2023-10-30T17:32:42.509 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:42.509 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:42.509 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:42.510 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:42.510 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:42.510 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:42.510 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:42.510 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:42.511 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:42.511 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:42.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] but i'm connected to v1:172.21.15.170:6789/0, reconnecting 2023-10-30T17:32:42.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:42.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 >> v1:172.21.15.170:6789/0 conn(0x7f8c78145be0 legacy=0x7f8c7813ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:42.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:32:42.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c780a9160 0x7f8c60003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:42.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient(hunting): picked mon.c con 0x7f8c780a9160 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:42.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c60003f40 0x7f8c600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:42.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient(hunting): picked mon.a con 0x7f8c60003f40 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:42.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c60004d00 0x7f8c60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:42.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient(hunting): picked mon.b con 0x7f8c60004d00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:42.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:42.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c780aa660 con 0x7f8c60003f40 2023-10-30T17:32:42.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:42.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c78145180 con 0x7f8c60004d00 2023-10-30T17:32:42.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:42.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6000bbc0 con 0x7f8c780a9160 2023-10-30T17:32:42.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:42.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.500+0000 7f8c75d9b640 10 monclient(hunting): _finish_auth 0 2023-10-30T17:32:42.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c76d9d640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c780a9160 0x7f8c60003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:42.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c7759e640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c60004d00 0x7f8c60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:42.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c76d9d640 10 monclient(hunting): get_auth_request con 0x7f8c780a9160 auth_method 0 2023-10-30T17:32:42.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:42.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c76d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:42.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:32:42.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c7759e640 10 monclient(hunting): get_auth_request con 0x7f8c60004d00 auth_method 0 2023-10-30T17:32:42.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:42.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c7759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:42.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c7759e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:32:42.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:42.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.503+0000 7f8c76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:42.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.504+0000 7f8c76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T17:32:42.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.504+0000 7f8c7759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:42.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.504+0000 7f8c7759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:42.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.504+0000 7f8c7759e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T17:32:42.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.504+0000 7f8c7659c640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c60003f40 0x7f8c600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:42.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7659c640 10 monclient(hunting): get_auth_request con 0x7f8c60003f40 auth_method 0 2023-10-30T17:32:42.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:42.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:42.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:32:42.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 10 monclient(hunting): handle_auth_done global_id 4374 payload 293 2023-10-30T17:32:42.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 1 -- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c780a9160 msgr2=0x7f8c60003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:42.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c780a9160 0x7f8c60003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 1 -- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c60003f40 msgr2=0x7f8c600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:42.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c60003f40 0x7f8c600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 10 monclient: _finish_hunting 0 2023-10-30T17:32:42.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 1 monclient: found mon.b 2023-10-30T17:32:42.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:42.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:42.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 1 -- 172.21.15.170:0/1115016669 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c60003520 con 0x7f8c60004d00 2023-10-30T17:32:42.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 10 monclient: _finish_auth 0 2023-10-30T17:32:42.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 10 monclient: _check_auth_tickets 2023-10-30T17:32:42.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7759e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:42.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7d24b640 5 monclient: authenticate success, global_id 4374 2023-10-30T17:32:42.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.505+0000 7f8c7d24b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:42.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.507+0000 7f8c7759e640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c60004d00 0x7f8c60005180 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8c78146030 tx=0x7f8c600057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49093dce574d990f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:42.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68034630 con 0x7f8c60004d00 2023-10-30T17:32:42.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:42.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c75d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:42.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c75d9b640 10 monclient: dump: 2023-10-30T17:32:42.527 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:42.528 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:42.528 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:42.528 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:42.528 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:42.528 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:42.528 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:42.529 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:42.529 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:42.529 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:42.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c7d24b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:32:42.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6802f070 con 0x7f8c60004d00 2023-10-30T17:32:42.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c75d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:42.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c7d24b640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:42.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c7d24b640 4 set_mon_vals no callback set 2023-10-30T17:32:42.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c75d9b640 1 -- 172.21.15.170:0/1115016669 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68034a50 con 0x7f8c60004d00 2023-10-30T17:32:42.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:42.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c75d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:42.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c75d9b640 10 monclient: dump: 2023-10-30T17:32:42.531 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:42.531 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:42.531 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:42.532 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:42.532 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:42.532 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:42.532 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:42.532 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:42.532 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:42.533 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:42.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c7d24b640 10 monclient: shutdown 2023-10-30T17:32:42.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c7d24b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:42.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1115016669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c60004d00 msgr2=0x7f8c60005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:42.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c7d24b640 1 --2- 172.21.15.170:0/1115016669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c60004d00 0x7f8c60005180 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8c78146030 tx=0x7f8c600057f0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c7759e640 1 -- 172.21.15.170:0/1115016669 reap_dead start 2023-10-30T17:32:42.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1115016669 shutdown_connections 2023-10-30T17:32:42.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.511+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1115016669 >> 172.21.15.170:0/1115016669 conn(0x7f8c780b72f0 msgr2=0x7f8c780b76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:42.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.512+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1115016669 shutdown_connections 2023-10-30T17:32:42.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.512+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1115016669 wait complete. 2023-10-30T17:32:42.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.512+0000 7f8c7d24b640 10 monclient: build_initial_monmap 2023-10-30T17:32:42.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.512+0000 7f8c7d24b640 10 monclient: monmap: 2023-10-30T17:32:42.535 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:42.535 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:42.535 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:42.535 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:42.536 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:42.536 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:42.536 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:42.536 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:42.536 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:42.537 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 librados: starting msgr at 2023-10-30T17:32:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 librados: starting objecter 2023-10-30T17:32:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 Processor -- start 2023-10-30T17:32:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 -- start start 2023-10-30T17:32:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 librados: setting wanted keys 2023-10-30T17:32:42.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 librados: calling monclient init 2023-10-30T17:32:42.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient: init 2023-10-30T17:32:42.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:42.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:32:42.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c60004d00 0x7f8c780ab060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient(hunting): picked mon.noname-c con 0x7f8c60004d00 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c78141e90 0x7f8c781da070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient(hunting): picked mon.noname-a con 0x7f8c78141e90 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c781da5b0 0x7f8c781d6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient(hunting): picked mon.noname-b con 0x7f8c781da5b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c781d9b50 con 0x7f8c78141e90 2023-10-30T17:32:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c781daa30 con 0x7f8c781da5b0 2023-10-30T17:32:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7813f810 con 0x7f8c60004d00 2023-10-30T17:32:42.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:42.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.513+0000 7f8c7d24b640 10 monclient(hunting): authenticate will time out at 1344.837991s 2023-10-30T17:32:42.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c76d9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c60004d00 0x7f8c780ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:42.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c7659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c78141e90 0x7f8c781da070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:42.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c7759e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c781da5b0 0x7f8c781d6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:42.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c76d9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c60004d00 0x7f8c780ab060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:35436/0 (socket says 172.21.15.170:35436) 2023-10-30T17:32:42.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c76d9d640 1 -- 172.21.15.170:0/1978620736 learned_addr learned my addr 172.21.15.170:0/1978620736 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:42.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c76d9d640 10 monclient(hunting): get_auth_request con 0x7f8c60004d00 auth_method 0 2023-10-30T17:32:42.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:42.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c76d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:42.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c76d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:42.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c7759e640 10 monclient(hunting): get_auth_request con 0x7f8c781da5b0 auth_method 0 2023-10-30T17:32:42.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:42.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c7759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:42.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c7759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:42.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c7659c640 10 monclient(hunting): get_auth_request con 0x7f8c78141e90 auth_method 0 2023-10-30T17:32:42.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:42.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c7659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:42.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.514+0000 7f8c7659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:42.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.515+0000 7f8c7759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:42.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.515+0000 7f8c7759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:42.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.515+0000 7f8c7759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:42.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.515+0000 7f8c76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:42.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.515+0000 7f8c76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:42.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.515+0000 7f8c76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:42.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.515+0000 7f8c7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:42.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.515+0000 7f8c7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:42.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.515+0000 7f8c7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:42.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.517+0000 7f8c7759e640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2023-10-30T17:32:42.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.517+0000 7f8c7759e640 1 -- 172.21.15.170:0/1978620736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c60004d00 msgr2=0x7f8c780ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:42.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.517+0000 7f8c7759e640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c60004d00 0x7f8c780ab060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.517+0000 7f8c7759e640 1 -- 172.21.15.170:0/1978620736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c78141e90 msgr2=0x7f8c781da070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:42.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.517+0000 7f8c7759e640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c78141e90 0x7f8c781da070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.517+0000 7f8c7759e640 10 monclient: _finish_hunting 0 2023-10-30T17:32:42.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.517+0000 7f8c7759e640 1 monclient: found mon.noname-b 2023-10-30T17:32:42.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.517+0000 7f8c7759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:42.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.517+0000 7f8c7759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:32:42.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.517+0000 7f8c7759e640 1 -- 172.21.15.170:0/1978620736 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c78141250 con 0x7f8c781da5b0 2023-10-30T17:32:42.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.520+0000 7f8c7759e640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c781da5b0 0x7f8c781d6650 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000bb90 tx=0x7f8c68001360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:42.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.520+0000 7f8c5f7fe640 10 client.?.objecter ms_handle_connect 0x7f8c781da5b0 2023-10-30T17:32:42.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.520+0000 7f8c5f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:42.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 1 -- 172.21.15.170:0/1978620736 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68034840 con 0x7f8c781da5b0 2023-10-30T17:32:42.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:42.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:42.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 10 monclient: dump: 2023-10-30T17:32:42.550 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:42.550 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:42.551 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:42.551 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:42.551 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:42.551 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:42.551 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:42.551 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:42.552 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:42.552 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:42.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 10 monclient: _finish_auth 0 2023-10-30T17:32:42.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:32:42.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:42.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 1 -- 172.21.15.170:0/1978620736 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c680349a0 con 0x7f8c781da5b0 2023-10-30T17:32:42.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c7d24b640 5 monclient: authenticate success, global_id 4375 2023-10-30T17:32:42.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c7d24b640 10 monclient: _renew_subs 2023-10-30T17:32:42.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c7d24b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:42.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1978620736 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c780ab640 con 0x7f8c781da5b0 2023-10-30T17:32:42.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:42.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 1 -- 172.21.15.170:0/1978620736 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68034d30 con 0x7f8c781da5b0 2023-10-30T17:32:42.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:42.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:32:42.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c5f7fe640 10 monclient: dump: 2023-10-30T17:32:42.554 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:42.555 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:42.555 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:42.555 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:42.555 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:42.555 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:42.555 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:42.556 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:42.556 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:42.556 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:42.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c7559a640 4 set_mon_vals no callback set 2023-10-30T17:32:42.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c7d24b640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:42.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c7d24b640 10 monclient: _renew_subs 2023-10-30T17:32:42.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c7d24b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:42.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1978620736 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c781d6d70 con 0x7f8c781da5b0 2023-10-30T17:32:42.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.523+0000 7f8c7d24b640 1 librados: init done 2023-10-30T17:32:42.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.524+0000 7f8c7d24b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:42.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.524+0000 7f8c7d24b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:42.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.524+0000 7f8c7d24b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:32:42.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.524+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1978620736 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7804e3c0 con 0x7f8c781da5b0 2023-10-30T17:32:42.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.761+0000 7f8c5f7fe640 1 -- 172.21.15.170:0/1978620736 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8c68031030 con 0x7f8c781da5b0 2023-10-30T17:32:42.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.762+0000 7f8c5f7fe640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8c6406c850 0x7f8c6406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:42.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.762+0000 7f8c76d9d640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8c6406c850 0x7f8c6406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:42.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.762+0000 7f8c76d9d640 10 monclient: get_auth_request con 0x7f8c6406c850 auth_method 0 2023-10-30T17:32:42.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.762+0000 7f8c76d9d640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8c6406c850 0x7f8c6406ed10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8c70006d40 tx=0x7f8c70009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:42.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.762+0000 7f8c5f7fe640 10 client.4375.objecter ms_handle_connect 0x7f8c6406c850 2023-10-30T17:32:42.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.763+0000 7f8c5f7fe640 1 -- 172.21.15.170:0/1978620736 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8c6807b220 con 0x7f8c781da5b0 2023-10-30T17:32:42.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.763+0000 7f8c5f7fe640 10 client.4375.objecter ms_dispatch 0x7f8c78000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:42.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.763+0000 7f8c5f7fe640 3 client.4375.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:42.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.763+0000 7f8c5f7fe640 3 client.4375.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:42.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.763+0000 7f8c5f7fe640 20 client.4375.objecter dump_active .. 0 homeless 2023-10-30T17:32:42.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.774+0000 7f8c5f7fe640 1 -- 172.21.15.170:0/1978620736 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8c68077560 con 0x7f8c781da5b0 2023-10-30T17:32:42.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.774+0000 7f8c5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:42.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.774+0000 7f8c5f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:42.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.940+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1978620736 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c781413e0 con 0x7f8c6406c850 2023-10-30T17:32:42.944 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.942+0000 7f8c5f7fe640 1 -- 172.21.15.170:0/1978620736 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47954 (secure 0 0 0) 0x7f8c781413e0 con 0x7f8c6406c850 2023-10-30T17:32:42.945 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:32:42.946 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-10-30T17:32:42.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 10 librados: watch_flush enter 2023-10-30T17:32:42.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 10 librados: watch_flush exit 2023-10-30T17:32:42.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 20 client.4375.objecter shutdown clearing up homeless session... 2023-10-30T17:32:42.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 10 client.4375.objecter successfully canceled tick 2023-10-30T17:32:42.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1978620736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8c6406c850 msgr2=0x7f8c6406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:42.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8c6406c850 0x7f8c6406ed10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8c70006d40 tx=0x7f8c70009190 comp rx=0 tx=0).stop 2023-10-30T17:32:42.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 10 monclient: shutdown 2023-10-30T17:32:42.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:42.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1978620736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c781da5b0 msgr2=0x7f8c781d6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:42.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c781da5b0 0x7f8c781d6650 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000bb90 tx=0x7f8c68001360 comp rx=0 tx=0).stop 2023-10-30T17:32:42.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1978620736 shutdown_connections 2023-10-30T17:32:42.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8c6406c850 0x7f8c6406ed10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c781da5b0 0x7f8c781d6650 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c78141e90 0x7f8c781da070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 --2- 172.21.15.170:0/1978620736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c60004d00 0x7f8c780ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:42.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1978620736 >> 172.21.15.170:0/1978620736 conn(0x7f8c780b72f0 msgr2=0x7f8c78139a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:42.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1978620736 shutdown_connections 2023-10-30T17:32:42.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 -- 172.21.15.170:0/1978620736 wait complete. 2023-10-30T17:32:42.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:42.945+0000 7f8c7d24b640 1 librados: shutdown 2023-10-30T17:32:42.957 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-30T17:32:40.981348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219136,"kb_used_data":2576,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755584,"statfs":{"total":773094113280,"available":772869718016,"internally_reserved":0,"allocated":2637824,"data_stored":1440848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":59,"apply_latency_ms":59,"commit_latency_ns":59000000,"apply_latency_ns":59000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.032460"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.769859+0000","last_change":"2023-10-30T17:32:35.372482+0000","last_active":"2023-10-30T17:32:36.769859+0000","last_peered":"2023-10-30T17:32:36.769859+0000","last_clean":"2023-10-30T17:32:36.769859+0000","last_became_active":"2023-10-30T17:32:35.367487+0000","last_became_peered":"2023-10-30T17:32:35.367487+0000","last_unstale":"2023-10-30T17:32:36.769859+0000","last_undegraded":"2023-10-30T17:32:36.769859+0000","last_fullsized":"2023-10-30T17:32:36.769859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:45:03.846674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.740287+0000","last_change":"2023-10-30T17:32:34.589393+0000","last_active":"2023-10-30T17:32:36.740287+0000","last_peered":"2023-10-30T17:32:36.740287+0000","last_clean":"2023-10-30T17:32:36.740287+0000","last_became_active":"2023-10-30T17:32:34.557021+0000","last_became_peered":"2023-10-30T17:32:34.557021+0000","last_unstale":"2023-10-30T17:32:36.740287+0000","last_undegraded":"2023-10-30T17:32:36.740287+0000","last_fullsized":"2023-10-30T17:32:36.740287+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:05.352859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.761366+0000","last_change":"2023-10-30T17:32:35.362652+0000","last_active":"2023-10-30T17:32:36.761366+0000","last_peered":"2023-10-30T17:32:36.761366+0000","last_clean":"2023-10-30T17:32:36.761366+0000","last_became_active":"2023-10-30T17:32:35.357489+0000","last_became_peered":"2023-10-30T17:32:35.357489+0000","last_unstale":"2023-10-30T17:32:36.761366+0000","last_undegraded":"2023-10-30T17:32:36.761366+0000","last_fullsized":"2023-10-30T17:32:36.761366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:21:36.073054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.743877+0000","last_change":"2023-10-30T17:32:34.658442+0000","last_active":"2023-10-30T17:32:36.743877+0000","last_peered":"2023-10-30T17:32:36.743877+0000","last_clean":"2023-10-30T17:32:36.743877+0000","last_became_active":"2023-10-30T17:32:34.631494+0000","last_became_peered":"2023-10-30T17:32:34.631494+0000","last_unstale":"2023-10-30T17:32:36.743877+0000","last_undegraded":"2023-10-30T17:32:36.743877+0000","last_fullsized":"2023-10-30T17:32:36.743877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:36:36.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.753615+0000","last_change":"2023-10-30T17:32:35.388556+0000","last_active":"2023-10-30T17:32:36.753615+0000","last_peered":"2023-10-30T17:32:36.753615+0000","last_clean":"2023-10-30T17:32:36.753615+0000","last_became_active":"2023-10-30T17:32:35.383414+0000","last_became_peered":"2023-10-30T17:32:35.383414+0000","last_unstale":"2023-10-30T17:32:36.753615+0000","last_undegraded":"2023-10-30T17:32:36.753615+0000","last_fullsized":"2023-10-30T17:32:36.753615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:46.189212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.612306+0000","last_change":"2023-10-30T17:32:34.318569+0000","last_active":"2023-10-30T17:32:36.612306+0000","last_peered":"2023-10-30T17:32:36.612306+0000","last_clean":"2023-10-30T17:32:36.612306+0000","last_became_active":"2023-10-30T17:32:34.287031+0000","last_became_peered":"2023-10-30T17:32:34.287031+0000","last_unstale":"2023-10-30T17:32:36.612306+0000","last_undegraded":"2023-10-30T17:32:36.612306+0000","last_fullsized":"2023-10-30T17:32:36.612306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:55:59.080654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.748690+0000","last_change":"2023-10-30T17:32:35.307241+0000","last_active":"2023-10-30T17:32:36.748690+0000","last_peered":"2023-10-30T17:32:36.748690+0000","last_clean":"2023-10-30T17:32:36.748690+0000","last_became_active":"2023-10-30T17:32:35.277163+0000","last_became_peered":"2023-10-30T17:32:35.277163+0000","last_unstale":"2023-10-30T17:32:36.748690+0000","last_undegraded":"2023-10-30T17:32:36.748690+0000","last_fullsized":"2023-10-30T17:32:36.748690+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:02.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:36.758421+0000","last_change":"2023-10-30T17:32:35.628758+0000","last_active":"2023-10-30T17:32:36.758421+0000","last_peered":"2023-10-30T17:32:36.758421+0000","last_clean":"2023-10-30T17:32:36.758421+0000","last_became_active":"2023-10-30T17:32:35.623395+0000","last_became_peered":"2023-10-30T17:32:35.623395+0000","last_unstale":"2023-10-30T17:32:36.758421+0000","last_undegraded":"2023-10-30T17:32:36.758421+0000","last_fullsized":"2023-10-30T17:32:36.758421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:47:08.984576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":118,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-30T17:32:38.071164+0000","last_change":"2023-10-30T17:32:28.719757+0000","last_active":"2023-10-30T17:32:38.071164+0000","last_peered":"2023-10-30T17:32:38.071164+0000","last_clean":"2023-10-30T17:32:38.071164+0000","last_became_active":"2023-10-30T17:32:28.686958+0000","last_became_peered":"2023-10-30T17:32:28.686958+0000","last_unstale":"2023-10-30T17:32:38.071164+0000","last_undegraded":"2023-10-30T17:32:38.071164+0000","last_fullsized":"2023-10-30T17:32:38.071164+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:25.778908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:25.778908+0000","last_clean_scrub_stamp":"2023-10-30T17:32:25.778908+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:41:21.329138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8350000000000009}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.9}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3529999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4410000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.940999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.352}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.052}]}]},{"osd":3,"up_from":16,"seq":68719476740,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2690000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6950000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.6709999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.030999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.509}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.206}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.890000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":630784,"data_stored":478486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8220000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.613}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9429999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.236000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.908000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.757000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.033999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.702}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":48.518000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.151}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.707000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.319000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.138}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.446999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.834}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2309999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.103999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.907}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.866}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.91}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.655000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.198}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.045}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3579999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.68}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.725999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.093}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.552}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":80646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.622}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.945}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.460000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.099}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.335999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.287000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.542999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":630784,"data_stored":478486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0830000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8579999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.711}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.192}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.831}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.35}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.311999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T17:32:42.959 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T17:32:42.959 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T17:32:42.960 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T17:32:42.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T17:32:43.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.056+0000 7f70b4dc9640 10 monclient: get_monmap_and_config 2023-10-30T17:32:43.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.056+0000 7f70b4dc9640 10 monclient: build_initial_monmap 2023-10-30T17:32:43.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.056+0000 7f70b4dc9640 10 monclient: monmap: 2023-10-30T17:32:43.059 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:43.060 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:43.060 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:32:43.058624+0000 2023-10-30T17:32:43.060 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:32:43.058624+0000 2023-10-30T17:32:43.060 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:43.061 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:43.061 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:32:43.061 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:32:43.061 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:32:43.061 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:32:43.062 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:32:43.062 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:43.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 1 Processor -- start 2023-10-30T17:32:43.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 1 -- start start 2023-10-30T17:32:43.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 10 monclient: init 2023-10-30T17:32:43.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:43.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-10-30T17:32:43.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f70b00b4370 0x7f70b00b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:43.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 10 monclient(hunting): picked mon.noname-c con 0x7f70b00b4370 addr v2:172.21.15.177:3300/0 2023-10-30T17:32:43.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b013eb00 0x7f70b013ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:43.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 10 monclient(hunting): picked mon.noname-b con 0x7f70b013eb00 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:43.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70b00ae6a0 addr v1:172.21.15.177:6789/0 2023-10-30T17:32:43.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b00af220 con 0x7f70b00ae6a0 2023-10-30T17:32:43.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:43.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b00b4750 con 0x7f70b00b4370 2023-10-30T17:32:43.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:43.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.057+0000 7f70b4dc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b00b02c0 con 0x7f70b013eb00 2023-10-30T17:32:43.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.058+0000 7f70b4dc9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:43.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.058+0000 7f70b4dc9640 10 monclient(hunting): authenticate will time out at 1345.382073s 2023-10-30T17:32:43.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.060+0000 7f70a5d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b013eb00 0x7f70b013ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:43.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.060+0000 7f70a5d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b013eb00 0x7f70b013ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:35440/0 (socket says 172.21.15.170:35440) 2023-10-30T17:32:43.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.060+0000 7f70a5d9b640 1 -- 172.21.15.170:0/4147977415 learned_addr learned my addr 172.21.15.170:0/4147977415 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:43.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.060+0000 7f70a5d9b640 10 monclient(hunting): get_auth_request con 0x7f70b013eb00 auth_method 0 2023-10-30T17:32:43.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.060+0000 7f70a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:43.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.060+0000 7f70a5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:43.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.060+0000 7f70a5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:43.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.061+0000 7f70a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:43.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.061+0000 7f70a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:43.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.061+0000 7f70a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:43.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.062+0000 7f70a5d9b640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2023-10-30T17:32:43.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.062+0000 7f70a5d9b640 1 -- 172.21.15.170:0/4147977415 >> v2:172.21.15.177:3300/0 conn(0x7f70b00b4370 msgr2=0x7f70b00b09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:43.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.062+0000 7f70a5d9b640 1 --2- 172.21.15.170:0/4147977415 >> v2:172.21.15.177:3300/0 conn(0x7f70b00b4370 0x7f70b00b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:43.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.062+0000 7f70a5d9b640 1 -- 172.21.15.170:0/4147977415 >> v1:172.21.15.177:6789/0 conn(0x7f70b00ae6a0 legacy=0x7f70b00aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:32:43.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.062+0000 7f70a5d9b640 10 monclient: _finish_hunting 0 2023-10-30T17:32:43.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.062+0000 7f70a5d9b640 1 monclient: found mon.noname-b 2023-10-30T17:32:43.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.062+0000 7f70a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:43.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.062+0000 7f70a5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:32:43.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.062+0000 7f70a5d9b640 1 -- 172.21.15.170:0/4147977415 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b00b05a0 con 0x7f70b013eb00 2023-10-30T17:32:43.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.065+0000 7f70a5d9b640 1 --2- 172.21.15.170:0/4147977415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b013eb00 0x7f70b013ef20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7098009870 tx=0x7f709802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87684f5184c1827 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:43.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 1 -- 172.21.15.170:0/4147977415 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70980317c0 con 0x7f70b013eb00 2023-10-30T17:32:43.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:43.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:32:43.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 10 monclient: dump: 2023-10-30T17:32:43.073 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:43.074 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:43.074 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:43.074 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:43.074 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:43.074 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:43.075 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:43.075 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:43.075 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:43.075 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:43.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 10 monclient: _finish_auth 0 2023-10-30T17:32:43.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 10 monclient: _check_auth_tickets 2023-10-30T17:32:43.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:43.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 1 -- 172.21.15.170:0/4147977415 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7098031920 con 0x7f70b013eb00 2023-10-30T17:32:43.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:43.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 1 -- 172.21.15.170:0/4147977415 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098031cb0 con 0x7f70b013eb00 2023-10-30T17:32:43.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:43.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:43.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70add9b640 10 monclient: dump: 2023-10-30T17:32:43.077 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:43.077 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:43.077 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:43.077 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:43.078 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:43.078 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:43.078 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:43.078 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:43.078 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:43.079 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:43.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70b4dc9640 5 monclient: authenticate success, global_id 4337 2023-10-30T17:32:43.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70b4dc9640 10 monclient: get_monmap_and_config success 2023-10-30T17:32:43.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.068+0000 7f70b4dc9640 4 set_mon_vals no callback set 2023-10-30T17:32:43.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.069+0000 7f70b4dc9640 10 monclient: shutdown 2023-10-30T17:32:43.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.069+0000 7f70b4dc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:43.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.069+0000 7f70b4dc9640 1 -- 172.21.15.170:0/4147977415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b013eb00 msgr2=0x7f70b013ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:43.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.069+0000 7f70b4dc9640 1 --2- 172.21.15.170:0/4147977415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b013eb00 0x7f70b013ef20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7098009870 tx=0x7f709802b860 comp rx=0 tx=0).stop 2023-10-30T17:32:43.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.069+0000 7f70b4dc9640 1 -- 172.21.15.170:0/4147977415 shutdown_connections 2023-10-30T17:32:43.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.069+0000 7f70b4dc9640 1 --2- 172.21.15.170:0/4147977415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b013eb00 0x7f70b013ef20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:43.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.069+0000 7f70b4dc9640 1 --2- 172.21.15.170:0/4147977415 >> v2:172.21.15.177:3300/0 conn(0x7f70b00b4370 0x7f70b00b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:43.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.069+0000 7f70b4dc9640 1 -- 172.21.15.170:0/4147977415 >> 172.21.15.170:0/4147977415 conn(0x7f70b0071950 msgr2=0x7f70b0071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:43.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.069+0000 7f70b4dc9640 1 -- 172.21.15.170:0/4147977415 shutdown_connections 2023-10-30T17:32:43.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.069+0000 7f70b4dc9640 1 -- 172.21.15.170:0/4147977415 wait complete. 2023-10-30T17:32:43.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.070+0000 7f70b4dc9640 10 monclient: build_initial_monmap 2023-10-30T17:32:43.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.070+0000 7f70b4dc9640 10 monclient: monmap: 2023-10-30T17:32:43.082 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:32:43.082 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:32:43.082 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:32:43.082 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:32:43.083 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:32:43.083 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:43.083 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:32:43.083 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:32:43.083 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:32:43.084 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:43.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.070+0000 7f70b4dc9640 1 librados: starting msgr at 2023-10-30T17:32:43.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.070+0000 7f70b4dc9640 1 librados: starting objecter 2023-10-30T17:32:43.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.070+0000 7f70b4dc9640 1 Processor -- start 2023-10-30T17:32:43.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 1 -- start start 2023-10-30T17:32:43.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 1 librados: setting wanted keys 2023-10-30T17:32:43.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 1 librados: calling monclient init 2023-10-30T17:32:43.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient: init 2023-10-30T17:32:43.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient: _reopen_session rank -1 2023-10-30T17:32:43.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:32:43.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70b00ae6a0 0x7f70b01d6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:43.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient(hunting): picked mon.noname-a con 0x7f70b00ae6a0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:32:43.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b00b4370 0x7f70b01d7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:43.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient(hunting): picked mon.noname-c con 0x7f70b00b4370 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:32:43.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70b013eb00 0x7f70b01427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:43.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient(hunting): picked mon.noname-b con 0x7f70b013eb00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:32:43.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:43.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b01d6860 con 0x7f70b00ae6a0 2023-10-30T17:32:43.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:43.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b0141260 con 0x7f70b013eb00 2023-10-30T17:32:43.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:32:43.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b0144900 con 0x7f70b00b4370 2023-10-30T17:32:43.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient(hunting): _renew_subs 2023-10-30T17:32:43.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70b4dc9640 10 monclient(hunting): authenticate will time out at 1345.395629s 2023-10-30T17:32:43.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.071+0000 7f70a5d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b00b4370 0x7f70b01d7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:43.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.072+0000 7f70a5d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b00b4370 0x7f70b01d7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:35450/0 (socket says 172.21.15.170:35450) 2023-10-30T17:32:43.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.072+0000 7f70a5d9b640 1 -- 172.21.15.170:0/3880798255 learned_addr learned my addr 172.21.15.170:0/3880798255 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:32:43.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.072+0000 7f70a5d9b640 10 monclient(hunting): get_auth_request con 0x7f70b00b4370 auth_method 0 2023-10-30T17:32:43.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.072+0000 7f70a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:43.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.072+0000 7f70a5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:43.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.072+0000 7f70a5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:43.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.072+0000 7f70a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:43.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.072+0000 7f70a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:43.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.072+0000 7f70a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:43.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.073+0000 7f70aed9d640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70b013eb00 0x7f70b01427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:43.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.074+0000 7f70aed9d640 10 monclient(hunting): get_auth_request con 0x7f70b013eb00 auth_method 0 2023-10-30T17:32:43.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.074+0000 7f70aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:32:43.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.074+0000 7f70aed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:32:43.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.074+0000 7f70aed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:32:43.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.074+0000 7f70aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:32:43.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.074+0000 7f70aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:32:43.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.074+0000 7f70aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:32:43.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.075+0000 7f70a5d9b640 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2023-10-30T17:32:43.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.075+0000 7f70a5d9b640 1 -- 172.21.15.170:0/3880798255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70b013eb00 msgr2=0x7f70b01427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:43.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.075+0000 7f70a5d9b640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70b013eb00 0x7f70b01427b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:43.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.075+0000 7f70a5d9b640 1 -- 172.21.15.170:0/3880798255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70b00ae6a0 msgr2=0x7f70b01d6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:43.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.075+0000 7f70a5d9b640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70b00ae6a0 0x7f70b01d6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:43.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.075+0000 7f70a5d9b640 10 monclient: _finish_hunting 0 2023-10-30T17:32:43.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.075+0000 7f70a5d9b640 1 monclient: found mon.noname-c 2023-10-30T17:32:43.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.075+0000 7f70a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:32:43.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.075+0000 7f70a5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:32:43.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.075+0000 7f70a5d9b640 1 -- 172.21.15.170:0/3880798255 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70980094d0 con 0x7f70b00b4370 2023-10-30T17:32:43.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.078+0000 7f70a5d9b640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b00b4370 0x7f70b01d7290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7098000c00 tx=0x7f70980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:43.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.078+0000 7f70a77fe640 10 client.?.objecter ms_handle_connect 0x7f70b00b4370 2023-10-30T17:32:43.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.078+0000 7f70a77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T17:32:43.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 1 -- 172.21.15.170:0/3880798255 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098032960 con 0x7f70b00b4370 2023-10-30T17:32:43.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:43.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:32:43.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 10 monclient: dump: 2023-10-30T17:32:43.098 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:43.098 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:43.098 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:43.098 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:43.099 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:43.099 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:43.099 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:43.099 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:43.099 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:43.099 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:43.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 10 monclient: _finish_auth 0 2023-10-30T17:32:43.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 10 monclient: _check_auth_tickets 2023-10-30T17:32:43.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:32:43.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 1 -- 172.21.15.170:0/3880798255 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7098032b00 con 0x7f70b00b4370 2023-10-30T17:32:43.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:32:43.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 1 -- 172.21.15.170:0/3880798255 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70980453f0 con 0x7f70b00b4370 2023-10-30T17:32:43.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 5 monclient: authenticate success, global_id 4340 2023-10-30T17:32:43.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 10 monclient: _renew_subs 2023-10-30T17:32:43.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:43.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 1 -- 172.21.15.170:0/3880798255 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b0144b20 con 0x7f70b00b4370 2023-10-30T17:32:43.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70ad59a640 4 set_mon_vals no callback set 2023-10-30T17:32:43.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:32:43.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:32:43.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70a77fe640 10 monclient: dump: 2023-10-30T17:32:43.102 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:32:43.103 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:32:43.103 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:43.103 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:32:43.103 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:32:43.103 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:32:43.103 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:32:43.104 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:32:43.105 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:32:43.105 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:32:43.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:32:43.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 10 monclient: _renew_subs 2023-10-30T17:32:43.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:43.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 1 -- 172.21.15.170:0/3880798255 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b0142ed0 con 0x7f70b00b4370 2023-10-30T17:32:43.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 1 librados: init done 2023-10-30T17:32:43.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:43.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:43.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:43.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.082+0000 7f70b4dc9640 1 -- 172.21.15.170:0/3880798255 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b0067cd0 con 0x7f70b00b4370 2023-10-30T17:32:43.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.321+0000 7f70a77fe640 1 -- 172.21.15.170:0/3880798255 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f709804e450 con 0x7f70b00b4370 2023-10-30T17:32:43.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.321+0000 7f70a77fe640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f709006c800 0x7f709006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:32:43.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.321+0000 7f70ae59c640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f709006c800 0x7f709006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:32:43.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.321+0000 7f70ae59c640 10 monclient: get_auth_request con 0x7f709006c800 auth_method 0 2023-10-30T17:32:43.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.322+0000 7f70ae59c640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f709006c800 0x7f709006ecc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009710 tx=0x7f70a0006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:32:43.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.322+0000 7f70a77fe640 10 client.4340.objecter ms_handle_connect 0x7f709006c800 2023-10-30T17:32:43.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.333+0000 7f70a77fe640 1 -- 172.21.15.170:0/3880798255 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f709807af90 con 0x7f70b00b4370 2023-10-30T17:32:43.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.333+0000 7f70a77fe640 10 client.4340.objecter ms_dispatch 0x7f70b0000b90 osd_map(23..23 src has 1..23) v4 2023-10-30T17:32:43.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.333+0000 7f70a77fe640 3 client.4340.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:32:43.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.333+0000 7f70a77fe640 3 client.4340.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:32:43.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.333+0000 7f70a77fe640 20 client.4340.objecter dump_active .. 0 homeless 2023-10-30T17:32:43.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.333+0000 7f70a77fe640 1 -- 172.21.15.170:0/3880798255 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7098076bd0 con 0x7f70b00b4370 2023-10-30T17:32:43.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.333+0000 7f70a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:32:43.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.333+0000 7f70a77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:32:43.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.551+0000 7f70b4dc9640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-10-30T17:32:43.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.551+0000 7f70b4dc9640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-10-30T17:32:43.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.551+0000 7f70b4dc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:32:43.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.552+0000 7f70b4dc9640 1 -- 172.21.15.170:0/3880798255 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f70b013e220 con 0x7f70b00b4370 2023-10-30T17:32:43.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.559+0000 7f70a77fe640 1 -- 172.21.15.170:0/3880798255 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f709807a1f0 con 0x7f70b00b4370 2023-10-30T17:32:43.562 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.559+0000 7f70a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-10-30T17:32:43.562 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.559+0000 7f70a77fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:32:43.562 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:32:43.562 INFO:teuthology.orchestra.run.smithi170.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T17:32:43.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.561+0000 7f70b4dc9640 10 librados: watch_flush enter 2023-10-30T17:32:43.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.561+0000 7f70b4dc9640 10 librados: watch_flush exit 2023-10-30T17:32:43.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.561+0000 7f70b4dc9640 20 client.4340.objecter shutdown clearing up homeless session... 2023-10-30T17:32:43.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.561+0000 7f70b4dc9640 10 client.4340.objecter successfully canceled tick 2023-10-30T17:32:43.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.561+0000 7f70b4dc9640 1 -- 172.21.15.170:0/3880798255 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f709006c800 msgr2=0x7f709006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:43.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.561+0000 7f70b4dc9640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f709006c800 0x7f709006ecc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009710 tx=0x7f70a0006c90 comp rx=0 tx=0).stop 2023-10-30T17:32:43.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.561+0000 7f70b4dc9640 10 monclient: shutdown 2023-10-30T17:32:43.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.561+0000 7f70b4dc9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:32:43.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.561+0000 7f70b4dc9640 1 -- 172.21.15.170:0/3880798255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b00b4370 msgr2=0x7f70b01d7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:32:43.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.561+0000 7f70b4dc9640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b00b4370 0x7f70b01d7290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7098000c00 tx=0x7f70980046c0 comp rx=0 tx=0).stop 2023-10-30T17:32:43.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.562+0000 7f70b4dc9640 1 -- 172.21.15.170:0/3880798255 shutdown_connections 2023-10-30T17:32:43.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.562+0000 7f70b4dc9640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f709006c800 0x7f709006ecc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:43.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.562+0000 7f70b4dc9640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70b013eb00 0x7f70b01427b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:43.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.562+0000 7f70b4dc9640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70b00b4370 0x7f70b01d7290 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:43.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.562+0000 7f70b4dc9640 1 --2- 172.21.15.170:0/3880798255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70b00ae6a0 0x7f70b01d6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:32:43.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.562+0000 7f70b4dc9640 1 -- 172.21.15.170:0/3880798255 >> 172.21.15.170:0/3880798255 conn(0x7f70b0071950 msgr2=0x7f70b00b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:32:43.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.562+0000 7f70b4dc9640 1 -- 172.21.15.170:0/3880798255 shutdown_connections 2023-10-30T17:32:43.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.562+0000 7f70b4dc9640 1 -- 172.21.15.170:0/3880798255 wait complete. 2023-10-30T17:32:43.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:32:43.562+0000 7f70b4dc9640 1 librados: shutdown 2023-10-30T17:32:43.575 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T17:32:43.576 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T17:32:43.592 INFO:tasks.workunit:Pulling workunits from ref fee4af45ddecf08e014b4503e218408d9e5c5be2 2023-10-30T17:32:43.593 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T17:32:43.594 DEBUG:teuthology.orchestra.run.smithi170:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T17:32:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T17:32:43.609 INFO:teuthology.orchestra.run.smithi170.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-30T17:32:43.609 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-30T17:32:43.664 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-30T17:32:43.664 DEBUG:teuthology.orchestra.run.smithi170:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-30T17:32:43.718 INFO:tasks.workunit:timeout=6h 2023-10-30T17:32:43.719 INFO:tasks.workunit:cleanup=True 2023-10-30T17:32:43.720 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout fee4af45ddecf08e014b4503e218408d9e5c5be2 2023-10-30T17:32:43.774 INFO:tasks.workunit.client.0.smithi170.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T17:32:56.463 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:01:40.429 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:56.518 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:01:40.484 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:56.557 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:01:40.522 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:56.563 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:40.502 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:56.571 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:40.510 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:56.572 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:01:40.538 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:56.579 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:01:40.545 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:56.609 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:40.548 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:32:56.623 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:01:40.589 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:16.534 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:02:00.499 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:16.617 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:02:00.583 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:16.640 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:02:00.606 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:16.660 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:00.599 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:16.666 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:00.604 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:16.680 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:00.619 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:16.694 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:02:00.660 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:16.698 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:02:00.665 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:16.722 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:02:00.688 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:32.678 INFO:tasks.workunit.client.0.smithi170.stderr:Note: switching to 'fee4af45ddecf08e014b4503e218408d9e5c5be2'. 2023-10-30T17:33:32.679 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-10-30T17:33:32.679 INFO:tasks.workunit.client.0.smithi170.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T17:33:32.679 INFO:tasks.workunit.client.0.smithi170.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T17:33:32.680 INFO:tasks.workunit.client.0.smithi170.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T17:33:32.680 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-10-30T17:33:32.680 INFO:tasks.workunit.client.0.smithi170.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T17:33:32.680 INFO:tasks.workunit.client.0.smithi170.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T17:33:32.680 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-10-30T17:33:32.680 INFO:tasks.workunit.client.0.smithi170.stderr: git switch -c 2023-10-30T17:33:32.681 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-10-30T17:33:32.681 INFO:tasks.workunit.client.0.smithi170.stderr:Or undo this operation with: 2023-10-30T17:33:32.681 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-10-30T17:33:32.681 INFO:tasks.workunit.client.0.smithi170.stderr: git switch - 2023-10-30T17:33:32.681 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-10-30T17:33:32.682 INFO:tasks.workunit.client.0.smithi170.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T17:33:32.682 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-10-30T17:33:32.682 INFO:tasks.workunit.client.0.smithi170.stderr:HEAD is now at fee4af45dde Merge branch 'bz_2241104' of https://github.com/ljflores/ceph into wip-yuri10-testing-2023-10-27-0804 2023-10-30T17:33:32.684 DEBUG:teuthology.orchestra.run.smithi170:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T17:33:32.749 INFO:tasks.workunit.client.0.smithi170.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T17:33:32.751 INFO:tasks.workunit.client.0.smithi170.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T17:33:32.752 INFO:tasks.workunit.client.0.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T17:33:33.398 INFO:tasks.workunit.client.0.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T17:33:33.470 INFO:tasks.workunit.client.0.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T17:33:33.509 INFO:tasks.workunit.client.0.smithi170.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T17:33:33.510 INFO:tasks.workunit.client.0.smithi170.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T17:33:33.510 INFO:tasks.workunit.client.0.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T17:33:33.548 INFO:tasks.workunit.client.0.smithi170.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T17:33:33.551 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-30T17:33:33.551 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T17:33:33.607 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-10-30T17:33:33.609 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-10-30T17:33:33.609 DEBUG:teuthology.orchestra.run.smithi170:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fee4af45ddecf08e014b4503e218408d9e5c5be2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-10-30T17:33:33.668 INFO:tasks.workunit.client.0.smithi170.stderr:+ parallel=1 2023-10-30T17:33:33.668 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' '' = --serial ']' 2023-10-30T17:33:33.668 INFO:tasks.workunit.client.0.smithi170.stderr:+ color= 2023-10-30T17:33:33.669 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -t 1 ']' 2023-10-30T17:33:33.669 INFO:tasks.workunit.client.0.smithi170.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-10-30T17:33:33.669 INFO:tasks.workunit.client.0.smithi170.stderr:+ declare -A pids 2023-10-30T17:33:33.669 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.669 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.670 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_aio 2023-10-30T17:33:33.670 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_aio' 2023-10-30T17:33:33.670 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_aio 2023-10-30T17:33:33.670 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.671 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_aio 2023-10-30T17:33:33.671 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91428 2023-10-30T17:33:33.671 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_aio on pid 91428' 2023-10-30T17:33:33.672 INFO:tasks.workunit.client.0.smithi170.stdout:test api_aio on pid 91428 2023-10-30T17:33:33.674 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91428 2023-10-30T17:33:33.675 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.675 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.675 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-10-30T17:33:33.675 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_aio_pp 2023-10-30T17:33:33.675 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_aio_pp' 2023-10-30T17:33:33.676 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_aio_pp 2023-10-30T17:33:33.676 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.676 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.676 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.676 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.676 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.677 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.677 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.677 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.677 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.677 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.678 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.678 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.678 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.678 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.678 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.679 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.679 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.679 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.679 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.679 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.679 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.680 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.680 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.680 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.680 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.680 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.681 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_aio_pp 2023-10-30T17:33:33.681 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91435 2023-10-30T17:33:33.681 INFO:tasks.workunit.client.0.smithi170.stdout:test api_aio_pp on pid 91435 2023-10-30T17:33:33.685 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_aio_pp on pid 91435' 2023-10-30T17:33:33.685 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91435 2023-10-30T17:33:33.685 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.685 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.686 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-10-30T17:33:33.686 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_io 2023-10-30T17:33:33.686 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_io' 2023-10-30T17:33:33.686 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_io 2023-10-30T17:33:33.686 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.687 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.687 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.687 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.687 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.687 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.687 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.688 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.688 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.688 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.688 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.688 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.689 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.689 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.689 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.689 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.689 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.690 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.690 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.690 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.690 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.690 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.690 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.691 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.691 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.691 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.691 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.691 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.692 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.692 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.692 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.692 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.692 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.692 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.693 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.693 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.693 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.693 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.693 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.694 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.694 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.694 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.694 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.694 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.694 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.695 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.695 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_io 2023-10-30T17:33:33.695 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91444 2023-10-30T17:33:33.695 INFO:tasks.workunit.client.0.smithi170.stdout:test api_io on pid 91444 2023-10-30T17:33:33.700 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_io on pid 91444' 2023-10-30T17:33:33.700 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91444 2023-10-30T17:33:33.701 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.701 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.701 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-10-30T17:33:33.701 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_io_pp 2023-10-30T17:33:33.701 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_io_pp' 2023-10-30T17:33:33.702 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_io_pp 2023-10-30T17:33:33.702 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.702 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.702 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.702 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.703 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.703 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:33.703 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.703 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.703 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.704 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:33.704 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.704 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:33.704 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.704 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:33.705 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:33.705 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.705 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.705 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.705 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.705 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.706 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.706 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.706 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.706 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.706 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.707 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.707 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.707 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.707 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.707 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.707 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.708 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.708 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.708 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.708 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.708 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.709 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.709 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.709 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.709 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.709 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.709 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.710 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.710 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.710 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.710 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.710 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.711 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.711 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.711 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.711 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.711 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.711 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.712 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.712 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.712 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_io_pp 2023-10-30T17:33:33.712 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91456 2023-10-30T17:33:33.713 INFO:tasks.workunit.client.0.smithi170.stdout:test api_io_pp on pid 91456 2023-10-30T17:33:33.718 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_io_pp on pid 91456' 2023-10-30T17:33:33.718 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91456 2023-10-30T17:33:33.718 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.718 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.719 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-10-30T17:33:33.719 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_asio 2023-10-30T17:33:33.719 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_asio' 2023-10-30T17:33:33.719 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_asio 2023-10-30T17:33:33.719 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:33.720 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.720 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:33.720 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:33.720 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.720 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:33.720 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.721 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:33.721 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.721 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.721 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:33.721 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.722 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:33.722 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.722 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:33.722 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.722 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.723 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.723 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.723 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.723 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:33.723 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.723 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:33.724 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:33.724 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.724 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.724 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.724 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.725 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.725 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.725 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.725 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.725 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.725 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.726 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:33.726 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.726 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.726 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.726 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.727 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.727 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.727 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.727 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.727 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.727 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.728 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.728 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.728 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.728 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.728 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.729 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.729 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.729 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.729 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:33.729 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:33.729 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.730 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.730 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.730 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:33.730 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.730 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.731 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.731 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:33.731 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.731 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:33.731 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.731 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.732 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.732 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.732 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.732 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.732 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.733 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.733 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.733 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:33.733 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.733 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:33.733 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:33.734 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_asio 2023-10-30T17:33:33.734 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:33.734 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91470 2023-10-30T17:33:33.734 INFO:tasks.workunit.client.0.smithi170.stdout:test api_asio on pid 91470 2023-10-30T17:33:33.739 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_asio on pid 91470' 2023-10-30T17:33:33.739 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91470 2023-10-30T17:33:33.740 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.740 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.740 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-10-30T17:33:33.740 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_list 2023-10-30T17:33:33.740 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_list' 2023-10-30T17:33:33.740 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_list 2023-10-30T17:33:33.741 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.741 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.741 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.741 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:33.742 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:33.742 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:33.743 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.743 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.743 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:33.743 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.743 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.744 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.744 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:33.744 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.744 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:33.744 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.744 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.745 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:33.745 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.745 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:33.745 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.745 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:33.746 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.746 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:33.746 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.746 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:33.746 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.747 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.747 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:33.747 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:33.747 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.747 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.747 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.748 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:33.748 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:33.748 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.748 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:33.748 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:33.749 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.749 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.749 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:33.749 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.749 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:33.749 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.750 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.750 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:33.750 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.750 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:33.750 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.751 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:33.751 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.751 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.751 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.751 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:33.752 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:33.752 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.752 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.752 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:33.752 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.752 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.753 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.753 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:33.753 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.753 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.753 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.753 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.754 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.754 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.754 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.754 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.754 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.755 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.755 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.755 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:33.755 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.755 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.756 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.756 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:33.756 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.756 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.756 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:33.756 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.757 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.757 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.757 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.757 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.757 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.757 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.758 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.758 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.758 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.758 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91428/exe 2023-10-30T17:33:33.758 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.759 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:33.759 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.759 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:33.759 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_list 2023-10-30T17:33:33.759 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91487 2023-10-30T17:33:33.759 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_list on pid 91487' 2023-10-30T17:33:33.760 INFO:tasks.workunit.client.0.smithi170.stdout:test api_list on pid 91487 2023-10-30T17:33:33.768 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91487 2023-10-30T17:33:33.768 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.768 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.768 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-10-30T17:33:33.769 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.769 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:33.769 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.769 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_lock 2023-10-30T17:33:33.769 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:33.770 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:33.770 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_lock' 2023-10-30T17:33:33.770 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:33.770 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:33.770 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:33.771 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_lock 2023-10-30T17:33:33.771 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:33.771 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:33.771 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:33.771 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:33.771 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:33.772 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.772 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:33.772 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:33.772 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:33.772 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.773 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.773 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:33.773 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.773 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_aio 2023-10-30T17:33:33.773 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:33.773 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:33.774 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.774 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.774 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.774 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.774 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_aio.log 2023-10-30T17:33:33.775 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:33.775 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.775 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.775 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:33.775 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:33.775 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:33.776 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:33.776 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_aio: /' 2023-10-30T17:33:33.776 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.776 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:33.776 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.777 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:33.777 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.777 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:33.777 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.777 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:33.777 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:33.778 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.778 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:33.778 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:33.778 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.778 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:33.779 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.779 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:33.779 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:33.779 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:33.779 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.779 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:33.780 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.780 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.780 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.780 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.780 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.781 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.781 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.781 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.781 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.781 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:33.781 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:33.782 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.782 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.782 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.782 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:33.782 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.783 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.783 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.783 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.783 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.783 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.783 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.784 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.784 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:33.784 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.784 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:33.784 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.785 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:33.785 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.785 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.785 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.785 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:33.785 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.786 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.786 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:33.786 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.786 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:33.786 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.787 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:33.787 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.787 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.787 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.787 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.787 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.788 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.788 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.788 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.788 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.788 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.789 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.789 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.789 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.789 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.789 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.789 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.790 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.790 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_lock 2023-10-30T17:33:33.790 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91504 2023-10-30T17:33:33.790 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_lock on pid 91504' 2023-10-30T17:33:33.791 INFO:tasks.workunit.client.0.smithi170.stdout:test api_lock on pid 91504 2023-10-30T17:33:33.799 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91504 2023-10-30T17:33:33.799 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.799 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.800 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91435/exe 2023-10-30T17:33:33.800 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-10-30T17:33:33.800 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_lock_pp 2023-10-30T17:33:33.800 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.800 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_lock_pp' 2023-10-30T17:33:33.801 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:33.801 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.801 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:33.801 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:33.801 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:33.802 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_lock_pp 2023-10-30T17:33:33.802 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:33.802 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:33.802 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.802 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.803 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:33.803 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.803 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:33.803 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.803 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:33.803 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.804 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.804 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.804 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.804 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.804 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:33.805 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:33.805 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:33.805 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:33.805 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.805 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:33.806 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.806 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:33.806 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:33.806 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.806 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.807 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:33.807 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.807 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:33.807 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:33.808 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:33.808 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:33.808 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:33.808 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.808 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:33.808 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.809 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:33.809 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:33.809 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:33.809 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:33.809 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:33.810 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.810 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:33.810 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.810 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:33.810 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:33.811 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.811 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.811 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.811 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.811 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.811 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:33.812 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.812 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:33.812 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.812 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:33.812 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.813 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.813 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.813 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:33.813 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.813 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.813 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.814 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.814 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:33.814 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.814 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.814 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.815 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:33.815 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.815 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.815 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.815 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.815 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:33.816 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.816 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.816 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.816 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.816 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.817 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:33.817 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.817 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.817 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.817 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.817 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:33.818 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.818 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:33.818 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:33.818 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.818 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.819 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.819 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.819 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.819 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.819 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.819 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:33.820 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.820 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:33.820 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.820 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:33.820 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.821 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.821 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-10-30T17:33:33.821 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.821 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.821 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91444/exe 2023-10-30T17:33:33.822 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.822 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.822 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_aio_pp: /' 2023-10-30T17:33:33.822 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.822 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:33.822 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.823 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:33.823 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:33.823 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_lock_pp 2023-10-30T17:33:33.823 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_aio_pp 2023-10-30T17:33:33.823 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:33.824 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91527 2023-10-30T17:33:33.824 INFO:tasks.workunit.client.0.smithi170.stdout:test api_lock_pp on pid 91527 2023-10-30T17:33:33.833 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_lock_pp on pid 91527' 2023-10-30T17:33:33.833 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91527 2023-10-30T17:33:33.833 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:33.833 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.833 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.834 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-10-30T17:33:33.834 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:33.834 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:33.834 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:33.834 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:33.835 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:33.835 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:33.835 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:33.835 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.835 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:33.836 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:33.836 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:33.836 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:33.836 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:33.836 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:33.836 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.837 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:33.837 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.837 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.837 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:33.837 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:33.838 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.838 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.838 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.838 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.838 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.838 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.839 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:33.839 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.839 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:33.839 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.839 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:33.840 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.840 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:33.840 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.840 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:33.840 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.841 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:33.841 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.841 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.841 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.841 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.841 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:33.842 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:33.842 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:33.842 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.842 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.842 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.843 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.843 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.843 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_io: /' 2023-10-30T17:33:33.843 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.843 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_io 2023-10-30T17:33:33.843 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.844 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.844 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.844 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:33.844 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.844 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.845 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:33.845 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.845 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:33.845 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_misc 2023-10-30T17:33:33.845 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.845 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.846 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.846 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.846 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:33.849 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.849 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.850 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:33.850 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.850 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:33.850 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.851 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.851 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.851 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.851 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:33.851 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_misc' 2023-10-30T17:33:33.851 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.852 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.852 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.852 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.852 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.852 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:33.853 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.853 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.853 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.853 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.853 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.854 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.854 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:33.854 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.854 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.854 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.854 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.855 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.855 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.855 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_io.log 2023-10-30T17:33:33.855 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:33.855 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_misc 2023-10-30T17:33:33.856 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.856 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91456/exe 2023-10-30T17:33:33.856 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.856 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:33.856 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.857 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:33.857 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:33.857 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.857 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:33.857 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:33.857 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.858 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:33.858 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.858 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.858 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:33.858 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:33.859 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:33.859 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.859 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:33.859 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:33.859 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.860 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:33.860 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.860 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:33.860 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:33.860 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:33.860 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_misc 2023-10-30T17:33:33.861 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.861 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91549 2023-10-30T17:33:33.861 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_misc on pid 91549' 2023-10-30T17:33:33.862 INFO:tasks.workunit.client.0.smithi170.stdout:test api_misc on pid 91549 2023-10-30T17:33:33.871 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91549 2023-10-30T17:33:33.872 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.872 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.872 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:33.872 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:33.873 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:33.873 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_misc_pp 2023-10-30T17:33:33.873 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:33.873 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:33.873 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:33.874 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:33.874 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_misc_pp' 2023-10-30T17:33:33.874 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:33.874 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:33.874 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:33.875 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:33.875 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:33.875 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:33.875 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.875 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:33.875 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.876 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:33.876 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.876 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.876 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:33.876 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_io_pp 2023-10-30T17:33:33.877 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.877 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-10-30T17:33:33.877 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.877 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.877 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:33.877 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.878 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:33.878 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.878 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:33.878 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:33.878 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.879 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.879 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.879 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.879 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.879 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.880 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:33.880 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.880 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:33.880 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.880 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:33.881 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_io_pp: /' 2023-10-30T17:33:33.881 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.881 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.881 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.881 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.882 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.882 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.882 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.882 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.882 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.882 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.883 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91470/exe 2023-10-30T17:33:33.883 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.883 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_misc_pp 2023-10-30T17:33:33.883 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.883 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.884 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.884 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.884 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.884 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.884 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:33.884 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.885 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.885 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.885 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.885 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.885 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.886 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.886 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:33.886 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:33.886 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:33.886 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.887 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:33.887 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:33.887 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:33.887 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.887 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:33.887 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.888 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.888 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:33.888 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:33.888 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:33.888 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.889 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.889 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.889 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.889 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.889 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.889 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.890 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.890 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:33.890 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.890 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:33.890 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:33.891 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.891 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:33.891 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_asio 2023-10-30T17:33:33.891 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.891 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:33.891 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:33.892 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.892 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:33.892 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.892 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.892 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:33.893 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:33.893 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_asio: /' 2023-10-30T17:33:33.893 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:33.893 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.893 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:33.893 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.894 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:33.894 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:33.894 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.894 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:33.894 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.895 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:33.895 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:33.895 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.895 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:33.895 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_misc_pp 2023-10-30T17:33:33.896 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91572 2023-10-30T17:33:33.896 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_misc_pp on pid 91572' 2023-10-30T17:33:33.896 INFO:tasks.workunit.client.0.smithi170.stdout:test api_misc_pp on pid 91572 2023-10-30T17:33:33.902 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91572 2023-10-30T17:33:33.902 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.903 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.903 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-10-30T17:33:33.903 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.903 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:33.903 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.904 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:33.904 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:33.904 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:33.904 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:33.904 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:33.905 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_tier_pp 2023-10-30T17:33:33.905 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:33.905 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.905 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:33.905 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_tier_pp' 2023-10-30T17:33:33.906 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.906 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:33.906 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:33.906 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-10-30T17:33:33.906 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.907 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:33.907 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.907 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:33.907 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.907 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.907 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:33.908 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.908 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:33.908 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:33.908 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:33.908 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:33.909 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:33.909 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.909 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.909 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.909 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:33.910 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.910 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.910 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.910 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.910 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.910 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:33.911 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:33.911 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.911 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.911 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:33.911 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.912 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:33.912 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.912 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:33.912 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:33.912 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.913 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:33.913 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.913 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:33.913 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:33.913 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_tier_pp 2023-10-30T17:33:33.914 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:33.914 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:33.914 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:33.914 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:33.914 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91487/exe 2023-10-30T17:33:33.914 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.915 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:33.915 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.915 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:33.915 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_tier_pp 2023-10-30T17:33:33.915 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:33.916 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.916 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:33.916 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.916 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.916 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:33.917 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91584 2023-10-30T17:33:33.917 INFO:tasks.workunit.client.0.smithi170.stdout:test api_tier_pp on pid 91584 2023-10-30T17:33:33.928 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_tier_pp on pid 91584' 2023-10-30T17:33:33.928 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91584 2023-10-30T17:33:33.928 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.928 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.929 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.929 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.929 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:33.929 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.929 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:33.930 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-10-30T17:33:33.930 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.930 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:33.930 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.930 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.931 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.931 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.931 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.931 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.931 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.931 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.932 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.932 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.932 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.932 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.932 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.933 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.933 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.933 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.933 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:33.933 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_asio.log 2023-10-30T17:33:33.933 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.934 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:33.934 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.934 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:33.934 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:33.934 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_pool 2023-10-30T17:33:33.935 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_pool' 2023-10-30T17:33:33.935 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:33.935 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:33.935 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:33.935 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:33.936 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:33.936 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:33.936 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:33.936 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.936 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:33.937 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:33.937 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.937 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:33.937 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:33.937 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.937 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_list.log 2023-10-30T17:33:33.938 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.938 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.938 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.938 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.938 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_list 2023-10-30T17:33:33.939 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.939 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.939 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.939 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.939 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.940 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.940 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.940 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.940 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.940 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.941 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.941 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.941 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.941 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.941 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.941 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.942 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.942 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.942 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.942 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.943 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:33.943 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:33.943 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:33.943 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.943 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:33.943 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.944 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:33.944 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:33.944 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.944 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:33.944 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:33.945 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.945 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:33.945 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.945 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:33.945 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:33.946 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.946 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:33.946 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.946 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:33.946 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:33.946 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_list: /' 2023-10-30T17:33:33.947 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.947 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:33.947 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91527/exe 2023-10-30T17:33:33.947 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:33.947 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.948 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:33.948 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.948 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:33.948 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:33.948 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.949 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:33.949 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:33.949 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:33.949 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:33.949 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:33.950 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:33.950 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.950 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.950 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:33.950 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_pool 2023-10-30T17:33:33.951 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:33.951 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.951 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:33.951 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:33.951 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:33.952 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.952 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.952 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.952 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.952 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.953 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.953 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.953 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_pool 2023-10-30T17:33:33.953 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.953 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.954 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.954 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91607 2023-10-30T17:33:33.954 INFO:tasks.workunit.client.0.smithi170.stdout:test api_pool on pid 91607 2023-10-30T17:33:33.965 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_pool on pid 91607' 2023-10-30T17:33:33.966 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91607 2023-10-30T17:33:33.966 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:33.966 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:33.966 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.967 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.967 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.967 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.967 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.967 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.968 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.968 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-10-30T17:33:33.968 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:33.968 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:33.968 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:33.969 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.969 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:33.969 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.969 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:33.969 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:33.969 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.970 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:33.970 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.970 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:33.970 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.970 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:33.971 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.971 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:33.971 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:33.971 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_lock_pp 2023-10-30T17:33:33.971 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.972 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.972 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.972 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.972 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.972 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.973 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.973 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.973 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:33.973 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:33.973 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.973 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.974 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.974 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.974 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:33.974 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:33.974 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.975 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:33.975 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.975 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.975 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.975 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.975 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.976 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.976 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.976 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:33.976 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_snapshots 2023-10-30T17:33:33.976 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:33.977 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_snapshots' 2023-10-30T17:33:33.977 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:33.977 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:33.977 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:33.977 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.978 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:33.978 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.978 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:33.978 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:33.978 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.979 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:33.979 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.979 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:33.979 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.979 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:33.979 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.980 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:33.980 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.980 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:33.980 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.980 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-10-30T17:33:33.981 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.981 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.981 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.981 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.981 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.982 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:33.982 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:33.982 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.982 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.982 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:33.982 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.983 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:33.983 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:33.983 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.983 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.983 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.984 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.984 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.984 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.984 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.984 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.985 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.985 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:33.985 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:33.985 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.985 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.985 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.986 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.986 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.986 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.986 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.986 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:33.987 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.987 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:33.987 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.987 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:33.987 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:33.988 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:33.988 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.988 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:33.988 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.988 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:33.988 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:33.989 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:33.989 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:33.989 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:33.989 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:33.989 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:33.990 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:33.990 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_lock_pp: /' 2023-10-30T17:33:33.990 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:33.990 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_snapshots 2023-10-30T17:33:33.990 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:33.990 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:33.991 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:33.991 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:33.991 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:33.991 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:33.991 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_snapshots 2023-10-30T17:33:33.992 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91654 2023-10-30T17:33:33.992 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_snapshots on pid 91654' 2023-10-30T17:33:33.992 INFO:tasks.workunit.client.0.smithi170.stdout:test api_snapshots on pid 91654 2023-10-30T17:33:33.999 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91654 2023-10-30T17:33:33.999 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.000 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.000 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_snapshots_pp 2023-10-30T17:33:34.000 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.000 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.000 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.000 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.001 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.001 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.001 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.001 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.001 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.002 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.002 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.002 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.002 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.002 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.003 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.003 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-10-30T17:33:34.003 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.003 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.003 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.004 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.004 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.004 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.004 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.004 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.004 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.005 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.005 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.005 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.005 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.005 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_snapshots_pp' 2023-10-30T17:33:34.006 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.006 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.006 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.006 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.006 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.007 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.007 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.007 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.007 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.007 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.008 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.008 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.008 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.008 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.008 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.008 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.009 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.009 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.009 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.009 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.009 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.010 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.010 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.010 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.010 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.010 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.011 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.011 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.011 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.011 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.011 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.012 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.012 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.012 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.012 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.012 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.012 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.013 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.013 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.013 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.013 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.013 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.014 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.014 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.014 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.014 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.014 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.014 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.015 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.015 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.015 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.015 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.015 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.016 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.016 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.016 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.016 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.016 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.017 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.017 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.017 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.017 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.017 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.017 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.018 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.018 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.018 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.018 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.018 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.019 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.019 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.019 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.019 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.019 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.020 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.020 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.020 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.020 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.020 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.021 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.021 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.021 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.021 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.021 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.021 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.022 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.022 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.022 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.022 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.022 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.023 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.023 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.023 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.023 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.023 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.024 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.024 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.024 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.024 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.024 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.024 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.025 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.025 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.025 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.025 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.025 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.026 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.026 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.026 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.026 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.026 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.027 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.027 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.027 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_snapshots_pp 2023-10-30T17:33:34.027 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.027 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.028 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.028 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.028 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.028 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.028 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.028 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.029 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.029 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.029 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.029 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.029 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.030 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.030 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.030 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.030 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.030 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.031 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.031 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.031 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.031 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.031 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.031 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.032 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.032 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.032 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.032 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.032 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_snapshots_pp 2023-10-30T17:33:34.033 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91688 2023-10-30T17:33:34.033 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_snapshots_pp on pid 91688' 2023-10-30T17:33:34.033 INFO:tasks.workunit.client.0.smithi170.stdout:test api_snapshots_pp on pid 91688 2023-10-30T17:33:34.050 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91688 2023-10-30T17:33:34.050 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.051 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.051 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.051 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.051 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.051 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.052 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.052 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91549/exe 2023-10-30T17:33:34.052 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.052 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.052 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.052 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.053 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.053 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.053 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.053 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.053 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.054 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.054 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.054 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.054 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.054 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.055 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.055 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.055 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.055 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.055 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.056 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.056 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.056 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.056 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.056 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.057 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.057 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.057 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.057 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.057 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.057 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.058 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.058 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.058 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.058 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.058 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.059 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91504/exe 2023-10-30T17:33:34.059 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.059 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.059 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.059 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.060 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91572/exe 2023-10-30T17:33:34.060 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.060 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.060 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.060 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.060 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.061 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.061 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.061 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.061 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.061 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.062 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.062 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.062 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.062 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.062 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.063 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.063 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.063 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.063 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.063 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.064 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.064 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.064 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.064 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.064 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.064 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-10-30T17:33:34.065 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.065 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.065 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.065 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_misc 2023-10-30T17:33:34.065 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.066 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.066 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.066 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.066 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.066 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_misc.log 2023-10-30T17:33:34.067 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.067 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.067 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.067 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.067 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.067 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.068 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.068 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_misc: /' 2023-10-30T17:33:34.068 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.068 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.068 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.069 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.069 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.069 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.069 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.069 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.070 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.070 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.070 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.070 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.070 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.071 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.071 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.071 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.071 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.071 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.071 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.072 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.072 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.072 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_stat 2023-10-30T17:33:34.072 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_misc_pp: /' 2023-10-30T17:33:34.072 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_stat' 2023-10-30T17:33:34.073 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_lock 2023-10-30T17:33:34.073 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_misc_pp 2023-10-30T17:33:34.073 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_lock: /' 2023-10-30T17:33:34.073 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.073 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.073 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.074 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.074 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.074 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.074 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.074 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.075 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.075 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.075 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.075 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.075 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.076 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.076 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.076 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.076 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.076 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.076 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.077 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.077 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.077 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.077 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.077 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.078 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.078 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.078 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.078 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.078 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.079 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.079 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.079 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.079 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.079 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.079 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.080 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.080 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.080 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.080 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.080 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.081 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91607/exe 2023-10-30T17:33:34.081 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.081 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.081 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.081 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.081 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.082 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.082 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.082 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.082 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.082 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.083 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.083 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_lock.log 2023-10-30T17:33:34.083 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.083 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.083 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.083 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.084 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.084 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.084 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.084 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.084 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.085 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.085 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.085 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.085 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.085 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_pool 2023-10-30T17:33:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_pool: /' 2023-10-30T17:33:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.087 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91584/exe 2023-10-30T17:33:34.087 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-10-30T17:33:34.087 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.087 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_stat 2023-10-30T17:33:34.087 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_pool.log 2023-10-30T17:33:34.088 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.088 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.088 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.088 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.088 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.088 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.089 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.089 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.089 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.089 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.089 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-10-30T17:33:34.090 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.090 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.090 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.090 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.090 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.091 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_tier_pp 2023-10-30T17:33:34.091 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.091 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.091 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_stat 2023-10-30T17:33:34.091 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91753 2023-10-30T17:33:34.092 INFO:tasks.workunit.client.0.smithi170.stdout:test api_stat on pid 91753 2023-10-30T17:33:34.096 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_stat on pid 91753' 2023-10-30T17:33:34.097 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91753 2023-10-30T17:33:34.097 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.097 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.097 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.097 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.098 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.098 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-10-30T17:33:34.098 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.098 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.098 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.098 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.099 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.099 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.099 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.099 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.099 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.100 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.100 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.100 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.100 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.100 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.101 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.101 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.101 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.101 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.101 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_tier_pp: /' 2023-10-30T17:33:34.102 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.102 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.102 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.102 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.102 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.103 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.103 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.103 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.103 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.103 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.104 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.104 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.104 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.104 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.104 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.104 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.105 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.105 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.105 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.105 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.105 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.106 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.106 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.106 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.106 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.107 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-10-30T17:33:34.107 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [----------] Global test environment set-up. 2023-10-30T17:33:34.107 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [----------] 12 tests from AsioRados 2023-10-30T17:33:34.107 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.783+0000 7fa659641780 10 monclient: get_monmap_and_config 2023-10-30T17:33:34.107 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.783+0000 7fa659641780 10 monclient: build_initial_monmap 2023-10-30T17:33:34.108 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.783+0000 7fa659641780 10 monclient: monmap: 2023-10-30T17:33:34.108 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: epoch 0 2023-10-30T17:33:34.108 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:33:34.108 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: last_changed 0.000000 2023-10-30T17:33:34.108 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: created 0.000000 2023-10-30T17:33:34.109 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: min_mon_release 0 (unknown) 2023-10-30T17:33:34.109 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: election_strategy: 1 2023-10-30T17:33:34.109 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:33:34.109 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:33:34.109 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:33:34.110 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.110 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.783+0000 7fa659641780 1 Processor -- start 2023-10-30T17:33:34.110 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 1 -- start start 2023-10-30T17:33:34.110 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient: init 2023-10-30T17:33:34.110 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient: _reopen_session rank -1 2023-10-30T17:33:34.111 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:33:34.111 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac96705b0 0x55aac9702710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:33:34.111 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient(hunting): picked mon.noname-a con 0x55aac96705b0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:33:34.111 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aac9672010 0x55aac96723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:33:34.111 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient(hunting): picked mon.noname-b con 0x55aac9672010 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:33:34.111 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac9704ae0 0x55aac96f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:33:34.112 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient(hunting): picked mon.noname-c con 0x55aac9704ae0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:33:34.112 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient(hunting): start opening mon connection 2023-10-30T17:33:34.112 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac9588320 con 0x55aac96705b0 2023-10-30T17:33:34.112 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient(hunting): start opening mon connection 2023-10-30T17:33:34.112 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac9587fc0 con 0x55aac9672010 2023-10-30T17:33:34.113 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient(hunting): start opening mon connection 2023-10-30T17:33:34.113 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aac95684e0 con 0x55aac9704ae0 2023-10-30T17:33:34.113 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient(hunting): _renew_subs 2023-10-30T17:33:34.113 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.784+0000 7fa659641780 10 monclient(hunting): authenticate will time out at 1396.108047s 2023-10-30T17:33:34.115 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.116 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.116 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.116 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.116 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.116 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.117 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.117 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.117 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.117 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.117 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.118 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.118 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.118 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.118 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.118 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.119 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.119 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.119 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.119 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.119 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_stat_pp 2023-10-30T17:33:34.120 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.790+0000 7fa6568cb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac96705b0 0x55aac9702710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:33:34.120 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.790+0000 7fa6568cb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac96705b0 0x55aac9702710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:53882/0 (socket says 172.21.15.170:53882) 2023-10-30T17:33:34.120 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.790+0000 7fa6568cb640 1 -- 172.21.15.170:0/694171111 learned_addr learned my addr 172.21.15.170:0/694171111 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:33:34.120 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.790+0000 7fa6568cb640 10 monclient(hunting): get_auth_request con 0x55aac96705b0 auth_method 0 2023-10-30T17:33:34.121 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.790+0000 7fa6568cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:33:34.121 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.790+0000 7fa6568cb640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:33:34.121 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.790+0000 7fa6568cb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:33:34.121 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.791+0000 7fa6568cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:33:34.121 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.791+0000 7fa6568cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:33:34.122 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.791+0000 7fa6568cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:33:34.122 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.792+0000 7fa6560ca640 1 --2- 172.21.15.170:0/694171111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aac9672010 0x55aac96723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:33:34.122 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6560ca640 10 monclient(hunting): get_auth_request con 0x55aac9672010 auth_method 0 2023-10-30T17:33:34.122 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6560ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:33:34.122 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6560ca640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:33:34.123 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6560ca640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:33:34.123 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6568cb640 10 monclient(hunting): handle_auth_done global_id 4392 payload 274 2023-10-30T17:33:34.123 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6568cb640 1 -- 172.21.15.170:0/694171111 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac9704ae0 msgr2=0x55aac96f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:33:34.123 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6568cb640 1 --2- 172.21.15.170:0/694171111 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac9704ae0 0x55aac96f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:33:34.123 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6568cb640 1 -- 172.21.15.170:0/694171111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aac9672010 msgr2=0x55aac96723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:33:34.124 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6568cb640 1 --2- 172.21.15.170:0/694171111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aac9672010 0x55aac96723f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:33:34.124 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6568cb640 10 monclient: _finish_hunting 0 2023-10-30T17:33:34.124 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6568cb640 1 monclient: found mon.noname-a 2023-10-30T17:33:34.130 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_stat_pp' 2023-10-30T17:33:34.130 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.130 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.131 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.131 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.131 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.131 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.131 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.132 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.132 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.132 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.132 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.132 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.133 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.133 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.133 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.133 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.133 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.133 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.134 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.134 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.134 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.134 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.134 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.135 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.135 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.135 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.135 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.135 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.136 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.136 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91654/exe 2023-10-30T17:33:34.136 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.136 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.136 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.137 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.137 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.137 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.137 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.137 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.137 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.138 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.138 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.138 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.138 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.138 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.139 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.139 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.139 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.139 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.139 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.140 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.140 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.140 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.140 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.140 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.140 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.141 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.141 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.141 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.141 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_snapshots 2023-10-30T17:33:34.141 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-10-30T17:33:34.142 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_snapshots: /' 2023-10-30T17:33:34.142 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.142 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_stat_pp 2023-10-30T17:33:34.142 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.142 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.143 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.143 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.143 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.143 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.143 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.143 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.144 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.144 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.144 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_stat_pp 2023-10-30T17:33:34.144 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91817 2023-10-30T17:33:34.144 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_stat_pp on pid 91817' 2023-10-30T17:33:34.145 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6568cb640 20 monclient:test api_stat_pp on pid 91817 2023-10-30T17:33:34.146 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91817 2023-10-30T17:33:34.147 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.147 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.147 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_watch_notify 2023-10-30T17:33:34.147 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_watch_notify' 2023-10-30T17:33:34.148 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.148 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-10-30T17:33:34.148 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.148 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.148 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.148 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_watch_notify 2023-10-30T17:33:34.149 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.149 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.149 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.149 INFO:tasks.workunit.client.0.smithi170.stdout: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:33:34.150 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6568cb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:33:34.150 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.793+0000 7fa6568cb640 1 -- 172.21.15.170:0/694171111 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa648005d20 con 0x55aac96705b0 2023-10-30T17:33:34.150 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.796+0000 7fa6568cb640 1 --2- 172.21.15.170:0/694171111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac96705b0 0x55aac9702710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa64c001e30 tx=0x7fa64c00b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=243351b739a6ec54 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:33:34.150 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 1 -- 172.21.15.170:0/694171111 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c014070 con 0x55aac96705b0 2023-10-30T17:33:34.150 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:33:34.151 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:33:34.151 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 10 monclient: dump: 2023-10-30T17:33:34.151 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: epoch 1 2023-10-30T17:33:34.151 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:33:34.151 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:33:34.152 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:33:34.152 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T17:33:34.152 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: election_strategy: 1 2023-10-30T17:33:34.152 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:33:34.152 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:33:34.153 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:33:34.153 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.153 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 10 monclient: _finish_auth 0 2023-10-30T17:33:34.153 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 10 monclient: _check_auth_tickets 2023-10-30T17:33:34.153 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:34.154 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa659641780 5 monclient: authenticate success, global_id 4392 2023-10-30T17:33:34.154 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa659641780 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:33:34.154 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 1 -- 172.21.15.170:0/694171111 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c008030 con 0x55aac96705b0 2023-10-30T17:33:34.154 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:33:34.154 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 1 -- 172.21.15.170:0/694171111 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00f400 con 0x55aac96705b0 2023-10-30T17:33:34.155 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:33:34.155 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:33:34.155 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa6550c8640 10 monclient: dump: 2023-10-30T17:33:34.155 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: epoch 1 2023-10-30T17:33:34.155 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:33:34.156 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:33:34.156 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:33:34.156 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T17:33:34.156 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: election_strategy: 1 2023-10-30T17:33:34.156 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:33:34.157 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:33:34.157 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:33:34.157 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_watch_notify 2023-10-30T17:33:34.158 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91830 2023-10-30T17:33:34.158 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_watch_notify on pid 91830' 2023-10-30T17:33:34.158 INFO:tasks.workunit.client.0.smithi170.stdout: atest api_watch_notify on pid 91830 2023-10-30T17:33:34.161 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91830 2023-10-30T17:33:34.161 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.162 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.162 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.162 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.162 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.162 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.163 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.163 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.163 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-10-30T17:33:34.163 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.163 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.163 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.164 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.164 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.164 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.164 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.164 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.165 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.165 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.165 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.165 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.165 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.166 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.166 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.166 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.166 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.166 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.167 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.167 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_watch_notify_pp 2023-10-30T17:33:34.167 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.167 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.167 INFO:tasks.workunit.client.0.smithi170.stdout:pi_asio: 2023-10-30T17:33:34.168 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa659641780 10 monclient: get_monmap_and_config success 2023-10-30T17:33:34.168 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.814+0000 7fa659641780 4 set_mon_vals no callback set 2023-10-30T17:33:34.168 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.815+0000 7fa659641780 10 monclient: shutdown 2023-10-30T17:33:34.168 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.815+0000 7fa659641780 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:33:34.168 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.815+0000 7fa659641780 1 -- 172.21.15.170:0/694171111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac96705b0 msgr2=0x55aac9702710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:33:34.169 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.815+0000 7fa659641780 1 --2- 172.21.15.170:0/694171111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac96705b0 0x55aac9702710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa64c001e30 tx=0x7fa64c00b670 comp rx=0 tx=0).stop 2023-10-30T17:33:34.169 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 1 -- 172.21.15.170:0/694171111 shutdown_connections 2023-10-30T17:33:34.169 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 1 --2- 172.21.15.170:0/694171111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aac9672010 0x55aac96723f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:33:34.169 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 1 --2- 172.21.15.170:0/694171111 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac9704ae0 0x55aac96f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:33:34.169 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 1 --2- 172.21.15.170:0/694171111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac96705b0 0x55aac9702710 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:33:34.170 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 1 -- 172.21.15.170:0/694171111 >> 172.21.15.170:0/694171111 conn(0x55aac9677730 msgr2=0x55aac9675920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:33:34.170 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 1 -- 172.21.15.170:0/694171111 shutdown_connections 2023-10-30T17:33:34.170 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 1 -- 172.21.15.170:0/694171111 wait complete. 2023-10-30T17:33:34.170 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 10 monclient: build_initial_monmap 2023-10-30T17:33:34.170 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 10 monclient: monmap: 2023-10-30T17:33:34.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: epoch 0 2023-10-30T17:33:34.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:33:34.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: last_changed 0.000000 2023-10-30T17:33:34.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: created 0.000000 2023-10-30T17:33:34.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: min_mon_release 0 (unknown) 2023-10-30T17:33:34.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: election_strategy: 1 2023-10-30T17:33:34.172 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:33:34.172 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:33:34.172 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:33:34.172 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.172 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 1 librados: starting msgr at 2023-10-30T17:33:34.173 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.816+0000 7fa659641780 1 librados: starting objecter 2023-10-30T17:33:34.173 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.817+0000 7fa659641780 1 Processor -- start 2023-10-30T17:33:34.173 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.817+0000 7fa659641780 1 -- start start 2023-10-30T17:33:34.173 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.817+0000 7fa659641780 1 librados: setting wanted keys 2023-10-30T17:33:34.173 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.817+0000 7fa659641780 1 librados: calling monclient init 2023-10-30T17:33:34.174 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.817+0000 7fa659641780 10 monclient: init 2023-10-30T17:33:34.174 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 10 monclient: _reopen_session rank -1 2023-10-30T17:33:34.174 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:33:34.179 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_watch_notify_pp' 2023-10-30T17:33:34.179 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.179 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.179 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.180 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.180 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_watch_notify_pp 2023-10-30T17:33:34.180 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.180 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.180 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.181 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.181 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.181 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.181 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.181 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.182 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.182 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.182 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.182 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.182 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.183 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.183 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.183 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.183 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.183 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.183 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.184 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.184 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.184 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.184 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.184 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.185 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.185 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.185 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.185 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.185 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.186 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.186 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.186 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.186 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.186 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.186 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.187 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.187 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.187 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.187 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.187 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.188 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.188 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.188 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.188 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.188 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.189 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.189 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.189 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.189 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.189 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.189 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.190 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.190 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.190 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.190 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.190 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.191 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.191 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.191 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.191 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_watch_notify_pp 2023-10-30T17:33:34.191 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.192 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.192 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.192 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.192 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.192 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.192 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.193 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.193 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.193 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.193 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.193 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.194 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91854 2023-10-30T17:33:34.194 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_watch_notify_pp on pid 91854' 2023-10-30T17:33:34.194 INFO:tasks.workunit.client.0.smithi170.stdout: api_asiotest api_watch_notify_pp on pid 91854 2023-10-30T17:33:34.200 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91854 2023-10-30T17:33:34.201 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.201 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.201 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.201 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.201 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.202 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.202 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.202 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.202 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.202 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.203 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91753/exe 2023-10-30T17:33:34.203 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.203 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.203 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.203 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.204 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.204 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.204 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.204 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.204 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.204 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_cmd 2023-10-30T17:33:34.205 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.205 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.205 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.205 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_cmd' 2023-10-30T17:33:34.205 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.206 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.206 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.206 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.206 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.206 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.206 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.207 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.207 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.207 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.207 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.207 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.208 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.208 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.208 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.208 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.208 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.209 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_stat: /' 2023-10-30T17:33:34.209 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.209 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.209 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.209 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.209 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.210 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.210 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.210 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.210 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.210 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.211 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.211 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.211 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.211 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.211 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.212 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.212 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.212 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.212 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_stat.log 2023-10-30T17:33:34.212 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_stat 2023-10-30T17:33:34.212 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_cmd 2023-10-30T17:33:34.213 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.213 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.213 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.213 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.213 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.214 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.214 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.214 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.214 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.214 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_cmd 2023-10-30T17:33:34.215 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.215 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.215 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.215 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.215 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91870 2023-10-30T17:33:34.216 INFO:tasks.workunit.client.0.smithi170.stdout:test api_cmd on pid 91870 2023-10-30T17:33:34.223 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_cmd on pid 91870' 2023-10-30T17:33:34.223 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91870 2023-10-30T17:33:34.223 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.224 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.224 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.224 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.224 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.224 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.224 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-10-30T17:33:34.225 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.225 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.225 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.225 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.225 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.226 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.226 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.226 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.226 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.226 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.227 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91688/exe 2023-10-30T17:33:34.227 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_cmd_pp 2023-10-30T17:33:34.227 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_cmd_pp' 2023-10-30T17:33:34.227 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.227 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_cmd_pp 2023-10-30T17:33:34.228 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-10-30T17:33:34.228 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.228 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.228 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.228 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.228 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.229 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.229 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.229 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.229 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.229 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.230 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.230 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.230 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.230 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.230 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.230 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.231 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_snapshots_pp 2023-10-30T17:33:34.231 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.231 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.231 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.231 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.232 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.232 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.232 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.232 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.232 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.232 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.233 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.233 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.233 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.233 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.233 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.234 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.234 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.234 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-10-30T17:33:34.234 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.234 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.235 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.235 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.235 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.235 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.235 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.236 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.236 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.236 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.236 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.236 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.236 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.237 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.237 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.237 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.237 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.237 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.238 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.238 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.238 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.238 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.238 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.239 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.239 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.239 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.239 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-10-30T17:33:34.239 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.240 INFO:tasks.workunit.client.0.smithi170.stdout:: 2023-10-30T17:33:33.818+0000 7fa659641780 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aac96705b0 0x55aac9639540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:33:34.240 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 10 monclient(hunting): picked mon.noname-b con 0x55aac96705b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:33:34.240 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac9672010 0x55aac96372c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:33:34.241 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 10 monclient(hunting): picked mon.noname-a con 0x55aac9672010 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:33:34.241 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac9704ae0 0x55aac9637800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:33:34.241 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 10 monclient(hunting): picked mon.noname-c con 0x55aac9704ae0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:33:34.241 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 10 monclient(hunting): start opening mon connection 2023-10-30T17:33:34.241 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac97a3f10 con 0x55aac9672010 2023-10-30T17:33:34.242 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 10 monclient(hunting): start opening mon connection 2023-10-30T17:33:34.242 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac97a41f0 con 0x55aac96705b0 2023-10-30T17:33:34.242 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 10 monclient(hunting): start opening mon connection 2023-10-30T17:33:34.242 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aac9637d40 con 0x55aac9704ae0 2023-10-30T17:33:34.242 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 10 monclient(hunting): _renew_subs 2023-10-30T17:33:34.243 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.818+0000 7fa659641780 10 monclient(hunting): authenticate will time out at 1396.141154s 2023-10-30T17:33:34.243 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.844+0000 7fa6570cc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac9704ae0 0x55aac9637800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:33:34.243 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.844+0000 7fa6568cb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aac96705b0 0x55aac9639540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:33:34.243 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.845+0000 7fa6570cc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac9704ae0 0x55aac9637800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:53204/0 (socket says 172.21.15.170:53204) 2023-10-30T17:33:34.243 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.845+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 learned_addr learned my addr 172.21.15.170:0/922528809 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:33:34.243 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.845+0000 7fa6570cc640 10 monclient(hunting): get_auth_request con 0x55aac9704ae0 auth_method 0 2023-10-30T17:33:34.244 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.845+0000 7fa6570cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:33:34.244 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.845+0000 7fa6570cc640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:33:34.245 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.245 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.246 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.246 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.246 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.246 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.247 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.247 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.247 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.247 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.247 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.247 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.248 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.845+0000 7fa6570cc640 10 moncltest api_cmd_pp on pid 91911 2023-10-30T17:33:34.261 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_cmd_pp 2023-10-30T17:33:34.261 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.261 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.262 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.262 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.262 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91911 2023-10-30T17:33:34.262 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_cmd_pp on pid 91911' 2023-10-30T17:33:34.262 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91911 2023-10-30T17:33:34.263 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.263 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.263 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.263 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.263 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.264 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.264 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.264 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.264 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.264 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.265 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.265 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.265 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.265 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.265 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-10-30T17:33:34.266 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.266 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.266 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.266 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.266 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.266 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.267 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.267 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.267 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.267 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.267 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.268 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.268 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.268 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.268 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.268 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.269 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.269 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.269 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.269 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.269 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.269 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91817/exe 2023-10-30T17:33:34.270 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.270 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.270 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.270 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.270 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.271 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.271 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.271 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.271 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.271 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.271 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.272 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.272 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.272 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.272 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.272 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.273 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.273 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.273 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.273 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.273 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.274 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.274 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.274 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.274 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.274 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.274 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.275 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.275 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.275 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.275 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.275 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.276 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.276 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.276 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.276 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.276 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.277 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-10-30T17:33:34.277 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.277 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.277 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.277 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.277 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.278 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.278 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.278 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.278 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.278 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.279 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.279 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.279 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.279 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.279 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.280 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.280 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.280 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.280 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.280 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.280 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.281 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.281 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.281 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.281 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.281 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.282 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.282 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.282 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.282 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.282 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.283 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.283 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.283 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.283 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.283 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.283 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.284 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.284 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.284 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.284 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.284 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.285 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_stat_pp: /' 2023-10-30T17:33:34.285 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_service 2023-10-30T17:33:34.285 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_service' 2023-10-30T17:33:34.285 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.285 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_stat_pp 2023-10-30T17:33:34.286 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_service 2023-10-30T17:33:34.286 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.286 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.286 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.286 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.286 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.287 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.287 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.287 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.287 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.287 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_service 2023-10-30T17:33:34.288 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91944 2023-10-30T17:33:34.288 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_service on pid 91944' 2023-10-30T17:33:34.288 INFO:tasks.workunit.client.0.smithi170.stdout:test api_service on pid 91944 2023-10-30T17:33:34.300 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=91944 2023-10-30T17:33:34.301 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.301 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.301 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-10-30T17:33:34.301 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.301 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.302 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.302 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.302 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.302 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.302 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.303 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.303 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.303 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.303 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.303 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.304 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.304 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.304 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.304 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.304 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.304 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.305 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.305 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.305 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.305 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.306 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.306 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.306 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.306 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.306 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.307 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.307 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.307 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.307 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.307 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.308 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.308 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.308 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.308 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.308 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.308 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.309 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.309 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.309 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.309 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.309 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.310 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_service_pp 2023-10-30T17:33:34.310 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_service_pp' 2023-10-30T17:33:34.310 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.310 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.310 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.310 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.311 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.311 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.311 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.311 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.311 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.312 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.312 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.312 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.312 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.312 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.312 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.313 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.313 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.313 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.313 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.313 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.314 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.314 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.314 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.314 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.314 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.315 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.315 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_service_pp 2023-10-30T17:33:34.315 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.315 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.315 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.315 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.316 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.316 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.316 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.316 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.316 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.317 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.317 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.317 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.317 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.317 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.317 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.318 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.318 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.318 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.318 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.318 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.318 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.319 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.319 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.319 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.319 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.319 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.320 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.320 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.320 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.320 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.320 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.320 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.321 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.321 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.321 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.321 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.321 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.322 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.322 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.322 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.322 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.322 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.323 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.323 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.323 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.323 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.323 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.323 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.324 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.324 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.324 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91830/exe 2023-10-30T17:33:34.324 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.324 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.325 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.325 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.325 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.325 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.325 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.325 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.326 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.326 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.326 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.326 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.326 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.327 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.327 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.327 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.327 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.327 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.328 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.328 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.328 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.328 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.328 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.328 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.329 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.329 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.329 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.329 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.329 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.330 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.330 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.330 INFO:tasks.workunit.client.0.smithi170.stdout:ient(hunting): _init_auth creating new auth 2023-10-30T17:33:34.330 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.845+0000 7fa6568cb640 10 monclient(hunting): get_auth_request con 0x55aac96705b0 auth_method 0 2023-10-30T17:33:34.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.845+0000 7fa6568cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:33:34.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.845+0000 7fa6568cb640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:33:34.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.845+0000 7fa6568cb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:33:34.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.846+0000 7fa6570cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:33:34.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.846+0000 7fa6570cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:33:34.332 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.846+0000 7fa6570cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:33:34.332 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.846+0000 7fa6568cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:33:34.332 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.846+0000 7fa6568cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:33:34.332 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.846+0000 7fa6568cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:33:34.332 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.850+0000 7fa6568cb640 10 monclient(hunting): handle_auth_done global_id 4414 payload 824 2023-10-30T17:33:34.333 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.850+0000 7fa6568cb640 1 -- 172.21.15.170:0/922528809 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac9704ae0 msgr2=0x55aac9637800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:33:34.333 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.850+0000 7fa6568cb640 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac9704ae0 0x55aac9637800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:33:34.333 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.850+0000 7fa6568cb640 1 -- 172.21.15.170:0/922528809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac9672010 msgr2=0x55aac96372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:33:34.333 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.850+0000 7fa6568cb640 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac9672010 0x55aac96372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:33:34.333 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.850+0000 7fa6568cb640 10 monclient: _finish_hunting 0 2023-10-30T17:33:34.334 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.850+0000 7fa6568cb640 1 monclient: found mon.noname-b 2023-10-30T17:33:34.334 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.850+0000 7fa6568cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:33:34.334 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.850+0000 7fa6568cb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:33:34.334 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.850+0000 7fa6568cb640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c00a040 con 0x55aac96705b0 2023-10-30T17:33:34.334 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.853+0000 7fa6568cb640 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aac96705b0 0x55aac9639540 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa64c013000 tx=0x7fa64c000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:33:34.335 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.853+0000 7fa63ffff640 10 client.?.objecter ms_handle_connect 0x55aac96705b0 2023-10-30T17:33:34.335 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.853+0000 7fa63ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T17:33:34.335 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c014040 con 0x55aac96705b0 2023-10-30T17:33:34.335 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.336 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:33:34.336 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:33:34.336 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 10 monclient: dump: 2023-10-30T17:33:34.336 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: epoch 1 2023-10-30T17:33:34.337 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:33:34.337 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:33:34.337 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:33:34.337 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T17:33:34.337 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: election_strategy: 1 2023-10-30T17:33:34.338 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:33:34.338 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:33:34.338 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:33:34.338 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.338 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 10 monclient: _finish_auth 0 2023-10-30T17:33:34.339 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 10 monclient: _check_auth_tickets 2023-10-30T17:33:34.339 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:34.339 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c001a60 con 0x55aac96705b0 2023-10-30T17:33:34.339 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:33:34.339 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c01b650 con 0x55aac96705b0 2023-10-30T17:33:34.340 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:33:34.340 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:33:34.340 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa63ffff640 10 monclient: dump: 2023-10-30T17:33:34.340 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: epoch 1 2023-10-30T17:33:34.340 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:33:34.340 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:33:34.341 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:33:34.341 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T17:33:34.341 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: election_strategy: 1 2023-10-30T17:33:34.341 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:33:34.341 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:33:34.342 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:33:34.342 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.342 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa659641780 5 monclient: authenticate success, global_id 4414 2023-10-30T17:33:34.342 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa659641780 10 monclient: _renew_subs 2023-10-30T17:33:34.342 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa659641780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:33:34.343 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55aac9594770 con 0x55aac96705b0 2023-10-30T17:33:34.343 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa659641780 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:33:34.343 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa659641780 10 monclient: _renew_subs 2023-10-30T17:33:34.343 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa659641780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:33:34.343 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aac9587920 con 0x55aac96705b0 2023-10-30T17:33:34.344 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa659641780 1 librados: init done 2023-10-30T17:33:34.352 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.352 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_watch_notify 2023-10-30T17:33:34.353 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.353 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.353 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.353 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.353 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.354 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-10-30T17:33:34.354 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.354 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.354 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.354 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.355 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.355 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.355 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.355 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.355 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.356 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.356 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.356 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.356 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.356 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.357 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.357 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.357 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.357 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.357 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.357 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.358 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.358 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.358 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.358 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.358 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.359 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.359 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.359 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.359 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.359 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.360 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.360 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.360 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.360 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.360 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.360 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.361 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.361 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.361 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.361 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.361 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.362 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.362 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.362 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.362 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.362 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.362 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.363 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.363 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.363 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.363 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.363 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.364 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.364 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.364 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.364 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.364 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.365 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.365 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.365 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.365 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.365 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.365 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.366 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_watch_notify: /' 2023-10-30T17:33:34.366 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.366 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.366 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.366 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.367 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.367 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.367 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.367 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.367 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.368 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.368 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.368 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.368 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.368 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.368 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.369 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.369 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.369 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.369 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.369 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.370 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.370 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.370 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.370 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.370 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.370 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.371 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.371 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.371 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.371 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.371 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91854/exe 2023-10-30T17:33:34.372 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.372 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91870/exe 2023-10-30T17:33:34.372 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_service_pp 2023-10-30T17:33:34.372 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92000 2023-10-30T17:33:34.373 INFO:tasks.workunit.client.0.smithi170.stdout: test api_service_pp on pid 92000 2023-10-30T17:33:34.379 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_service_pp on pid 92000' 2023-10-30T17:33:34.379 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=92000 2023-10-30T17:33:34.379 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.380 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.380 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_c_write_operations 2023-10-30T17:33:34.380 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_c_write_operations' 2023-10-30T17:33:34.380 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.380 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.380 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.381 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.381 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.381 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.381 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.381 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.382 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.382 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.382 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.382 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_c_write_operations 2023-10-30T17:33:34.382 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.383 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.383 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.383 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.383 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.383 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.384 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.384 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.384 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.384 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.384 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.385 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.385 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.385 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.385 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.385 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.385 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.386 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.386 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-10-30T17:33:34.386 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.386 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.386 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.387 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.387 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.387 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.387 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.387 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.388 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.388 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.388 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91944/exe 2023-10-30T17:33:34.388 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.388 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.389 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.389 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-10-30T17:33:34.389 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_watch_notify_pp 2023-10-30T17:33:34.389 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-10-30T17:33:34.389 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.389 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.390 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.390 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_cmd.log 2023-10-30T17:33:34.390 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_cmd 2023-10-30T17:33:34.390 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.390 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.391 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.391 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.391 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.391 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.391 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_cmd: /' 2023-10-30T17:33:34.392 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.392 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.392 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.392 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.392 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.392 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.393 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.393 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.393 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.393 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.393 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.394 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_c_write_operations 2023-10-30T17:33:34.394 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92021 2023-10-30T17:33:34.394 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_c_write_operations on pid 92021' 2023-10-30T17:33:34.394 INFO:tasks.workunit.client.0.smithi170.stdout:test api_c_write_operations on pid 92021 2023-10-30T17:33:34.406 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=92021 2023-10-30T17:33:34.407 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.407 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.407 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.407 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.408 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_service: /' 2023-10-30T17:33:34.408 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.408 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.408 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.408 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.408 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-10-30T17:33:34.409 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.409 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.409 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.409 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.409 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.410 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.410 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.410 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.410 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.410 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.411 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.411 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.411 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.411 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_service 2023-10-30T17:33:34.411 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.412 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.412 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.412 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.412 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.412 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.412 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.413 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.413 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.413 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.413 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.413 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.414 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.414 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.414 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.414 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.414 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.414 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.415 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.415 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.415 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.415 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.415 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.416 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.416 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.416 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.416 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.416 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.417 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.417 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.417 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.417 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.417 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.418 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.418 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.418 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.418 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.418 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.418 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.419 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.419 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.419 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.419 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.419 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.420 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.420 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.420 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.420 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.420 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.420 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.421 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.421 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.421 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.421 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.421 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.422 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.422 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.422 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.422 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.422 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.423 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.423 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.423 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.423 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.423 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.423 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.424 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.424 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.424 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.424 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.424 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.425 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.425 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.425 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.425 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.425 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.426 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.426 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.426 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.426 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.426 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.427 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.427 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.427 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.427 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.427 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.427 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.428 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.428 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.428 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.428 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.428 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.429 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.429 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.429 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.429 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.429 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.429 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.430 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.430 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.430 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.430 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.430 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.431 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.431 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.431 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.431 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.431 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.432 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.432 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_c_read_operations 2023-10-30T17:33:34.432 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.432 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.432 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.433 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_c_read_operations' 2023-10-30T17:33:34.433 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.433 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.433 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.433 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.433 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.434 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.434 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.434 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.434 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.434 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.435 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_c_read_operations 2023-10-30T17:33:34.435 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.435 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.435 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.435 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.436 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.436 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.436 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/91911/exe 2023-10-30T17:33:34.436 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_service.log 2023-10-30T17:33:34.436 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_c_read_operations 2023-10-30T17:33:34.437 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.437 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92055 2023-10-30T17:33:34.437 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_c_read_operations on pid 92055' 2023-10-30T17:33:34.437 INFO:tasks.workunit.client.0.smithi170.stdout:test api_c_read_operations on pid 92055 2023-10-30T17:33:34.445 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=92055 2023-10-30T17:33:34.446 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.446 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.446 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-10-30T17:33:34.446 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.447 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.447 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.447 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.447 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.447 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.448 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.448 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.448 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.448 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.448 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.449 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.449 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.449 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.449 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.449 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.450 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.450 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.450 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.450 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.450 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.450 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.451 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s api_cls_remote_reads 2023-10-30T17:33:34.451 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.451 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.451 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.451 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.452 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.452 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.452 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' api_cls_remote_reads' 2023-10-30T17:33:34.452 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.452 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.453 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.453 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.453 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.453 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.453 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.454 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.454 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.454 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-10-30T17:33:34.454 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.454 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.455 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.455 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.455 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.455 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_cmd_pp 2023-10-30T17:33:34.455 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-10-30T17:33:34.455 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.456 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.456 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.456 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.456 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.456 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.457 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.457 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.457 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.457 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.457 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.457 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.458 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.458 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.458 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.458 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.458 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.459 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.459 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.459 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.459 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo api_cls_remote_reads 2023-10-30T17:33:34.459 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.461 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.461 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.461 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.462 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.462 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.462 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.462 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.462 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.463 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.463 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.463 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.463 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.463 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.463 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.464 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.464 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.464 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.464 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.464 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.465 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.465 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.465 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.465 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.465 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=api_cls_remote_reads 2023-10-30T17:33:34.466 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.466 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.466 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.466 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.466 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.467 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.467 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92084 2023-10-30T17:33:34.467 INFO:tasks.workunit.client.0.smithi170.stdout:test api_cls_remote_reads on pid 92084 2023-10-30T17:33:34.474 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test api_cls_remote_reads on pid 92084' 2023-10-30T17:33:34.475 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=92084 2023-10-30T17:33:34.475 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.475 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.475 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.475 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.476 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.476 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.476 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.476 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.476 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.477 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.477 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.477 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.477 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.477 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s list_parallel 2023-10-30T17:33:34.478 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.478 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.478 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.478 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.478 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' list_parallel' 2023-10-30T17:33:34.479 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.479 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-10-30T17:33:34.479 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/92000/exe 2023-10-30T17:33:34.479 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.479 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.480 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.480 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.480 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.480 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.480 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.480 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.481 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.481 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.481 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.481 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.481 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.482 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.482 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.482 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.482 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.482 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.483 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.483 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.483 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.483 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.483 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.484 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.484 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.484 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.484 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.484 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.485 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.485 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.485 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.485 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.485 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.485 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.486 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.486 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.486 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-10-30T17:33:34.486 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_service_pp: /' 2023-10-30T17:33:34.486 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_service_pp 2023-10-30T17:33:34.487 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.487 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.487 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.487 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.487 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.487 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.488 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.488 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.488 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.488 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.488 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.489 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.489 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.489 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.489 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.489 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.489 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.490 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.490 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.490 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.490 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.490 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo list_parallel 2023-10-30T17:33:34.491 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=list_parallel 2023-10-30T17:33:34.491 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92101 2023-10-30T17:33:34.491 INFO:tasks.workunit.client.0.smithi170.stdout:test list_parallel on pid 92101 2023-10-30T17:33:34.495 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test list_parallel on pid 92101' 2023-10-30T17:33:34.495 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=92101 2023-10-30T17:33:34.496 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.496 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.496 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.496 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.496 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.497 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.497 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.497 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.497 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.497 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.498 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.498 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.498 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.498 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.498 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.498 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.499 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.499 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.499 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.499 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.499 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.500 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.500 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.500 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.500 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.500 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.501 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.501 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.501 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.501 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.501 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.501 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.502 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.502 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-10-30T17:33:34.502 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.502 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.502 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.503 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.503 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa659641780 10 librados: wait_for_osdmap waiting 2023-10-30T17:33:34.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.875+0000 7fa6550c8640 4 set_mon_vals no callback set 2023-10-30T17:33:34.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.928+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa64c01b7b0 con 0x55aac96705b0 2023-10-30T17:33:34.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.928+0000 7fa63ffff640 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa64406aa40 0x7fa64406cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:33:34.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.929+0000 7fa6560ca640 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa64406aa40 0x7fa64406cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:33:34.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.929+0000 7fa6560ca640 10 monclient: get_auth_request con 0x7fa64406aa40 auth_method 0 2023-10-30T17:33:34.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.929+0000 7fa6560ca640 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa64406aa40 0x7fa64406cf00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55aac970b6e0 tx=0x7fa65000b1b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:33:34.505 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.929+0000 7fa63ffff640 10 client.4414.objecter ms_handle_connect 0x7fa64406aa40 2023-10-30T17:33:34.505 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.929+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa64c058b80 con 0x55aac96705b0 2023-10-30T17:33:34.505 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.929+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(23..23 src has 1..23) v4 2023-10-30T17:33:34.505 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.929+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [23,23] > 0 2023-10-30T17:33:34.505 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.929+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding full epoch 23 2023-10-30T17:33:34.506 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.930+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:33:34.506 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.930+0000 7fa659641780 10 librados: wait_for_osdmap done waiting 2023-10-30T17:33:34.506 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.930+0000 7fa659641780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:33:34.506 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.930+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa64804ad70 con 0x55aac96705b0 2023-10-30T17:33:34.506 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.944+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa64c058250 con 0x55aac96705b0 2023-10-30T17:33:34.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.944+0000 7fa63ffff640 10 monclient: handle_get_version_reply finishing 1 version 23 2023-10-30T17:33:34.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.944+0000 7fa6548c7640 10 client.4414.objecter _get_latest_version latest 23, have it 2023-10-30T17:33:34.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.944+0000 7fa659641780 10 client.4414.objecter create_pool name=ceph_test_rados_api_asio 2023-10-30T17:33:34.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.944+0000 7fa659641780 10 client.4414.objecter pool_op_submit 1 2023-10-30T17:33:34.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.944+0000 7fa659641780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:33:34.519 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.519 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.519 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.519 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.520 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.520 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.520 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.520 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.520 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.521 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s open_pools_parallel 2023-10-30T17:33:34.521 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.521 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.521 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.521 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' open_pools_parallel' 2023-10-30T17:33:34.522 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.522 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.522 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.522 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.522 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.523 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.523 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.523 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.523 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.523 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.524 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.524 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.524 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.524 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.524 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.524 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.525 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.525 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.525 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.525 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.525 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.526 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.526 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.526 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.526 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.526 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.527 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.527 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.527 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.527 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.527 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo open_pools_parallel 2023-10-30T17:33:34.528 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.528 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.528 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.528 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.528 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.528 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.529 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.529 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.529 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.529 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.529 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.530 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.530 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.530 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.530 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.530 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.531 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.531 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.531 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.531 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.531 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/92021/exe 2023-10-30T17:33:34.532 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.532 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.532 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.532 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.532 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.532 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.533 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.533 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.533 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.533 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.533 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.534 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.534 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.534 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.534 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.534 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.535 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.535 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.535 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.535 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.535 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.536 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.536 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.536 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.536 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.536 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.536 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_c_write_operations 2023-10-30T17:33:34.537 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-10-30T17:33:34.537 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.537 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-10-30T17:33:34.537 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.537 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.538 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.538 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.538 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.538 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.538 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.539 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.539 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.539 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.539 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.539 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.539 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.540 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.540 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.540 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.540 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.540 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.541 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.541 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.541 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.541 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.541 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.542 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.542 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.542 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.542 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.542 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.543 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.543 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.543 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.543 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.543 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.544 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.544 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.544 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.544 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.544 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.544 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.545 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.545 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.545 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.545 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/92055/exe 2023-10-30T17:33:34.545 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=open_pools_parallel 2023-10-30T17:33:34.546 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92177 2023-10-30T17:33:34.546 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test open_pools_parallel on pid 92177' 2023-10-30T17:33:34.546 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:33.944+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55aac95f7e40 con 0x55test open_pools_parallel on pid 92177 2023-10-30T17:33:34.551 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=92177 2023-10-30T17:33:34.551 INFO:tasks.workunit.client.0.smithi170.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T17:33:34.552 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.552 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.552 INFO:tasks.workunit.client.0.smithi170.stderr:++ printf %25s delete_pools_parallel 2023-10-30T17:33:34.552 INFO:tasks.workunit.client.0.smithi170.stderr:+ r=' delete_pools_parallel' 2023-10-30T17:33:34.552 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.553 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.553 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.553 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.553 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.553 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-10-30T17:33:34.554 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.554 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.554 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.554 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.554 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.555 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.555 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.555 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.555 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.555 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.556 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.556 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.556 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.556 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.556 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.556 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.557 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.557 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.557 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.557 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.557 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.557 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.558 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_c_read_operations 2023-10-30T17:33:34.558 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo delete_pools_parallel 2023-10-30T17:33:34.558 INFO:tasks.workunit.client.0.smithi170.stderr:++ awk '{print $1}' 2023-10-30T17:33:34.558 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.558 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.559 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.559 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.559 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.559 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.559 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.559 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-10-30T17:33:34.560 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-10-30T17:33:34.560 INFO:tasks.workunit.client.0.smithi170.stderr:+ ff=delete_pools_parallel 2023-10-30T17:33:34.560 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.560 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.560 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.561 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.561 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.561 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.561 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.561 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.561 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.562 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.562 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.562 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.562 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.562 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.563 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.563 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.563 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.563 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.563 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92193 2023-10-30T17:33:34.564 INFO:tasks.workunit.client.0.smithi170.stdout:test delete_pools_parallel on pid 92193 2023-10-30T17:33:34.587 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo 'test delete_pools_parallel on pid 92193' 2023-10-30T17:33:34.587 INFO:tasks.workunit.client.0.smithi170.stderr:+ pids[$f]=92193 2023-10-30T17:33:34.588 INFO:tasks.workunit.client.0.smithi170.stderr:+ ret=0 2023-10-30T17:33:34.588 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.588 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.588 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T17:33:34.588 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.589 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:33:34.589 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91584 2023-10-30T17:33:34.589 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91584 2023-10-30T17:33:34.589 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.589 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.590 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.590 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.590 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.590 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.590 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.591 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.591 INFO:tasks.workunit.client.0.smithi170.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-10-30T17:33:34.591 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.591 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.591 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.591 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.592 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T17:33:34.592 INFO:tasks.workunit.client.0.smithi170.stderr:+ . /etc/bashrc 2023-10-30T17:33:34.592 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -z '' ']' 2023-10-30T17:33:34.592 INFO:tasks.workunit.client.0.smithi170.stderr:++ BASHRCSOURCED=Y 2023-10-30T17:33:34.592 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.593 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.593 INFO:tasks.workunit.client.0.smithi170.stderr:++ shopt -q login_shell 2023-10-30T17:33:34.593 INFO:tasks.workunit.client.0.smithi170.stderr:+++ umask 2023-10-30T17:33:34.593 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T17:33:34.594 INFO:tasks.workunit.client.0.smithi170.stderr:++ SHELL=/bin/bash 2023-10-30T17:33:34.594 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.594 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.594 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.594 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.595 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.595 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.595 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.595 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.595 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T17:33:34.595 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.596 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.596 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.596 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T17:33:34.596 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.596 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.596 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.597 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T17:33:34.597 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.597 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.597 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.597 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.598 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.598 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.598 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.598 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.598 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.599 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.599 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.599 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.599 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.599 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.600 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.600 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.600 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.600 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.600 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.601 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.601 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.601 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.601 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.601 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.601 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T17:33:34.602 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.602 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T17:33:34.602 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.602 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.602 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.603 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.603 INFO:tasks.workunit.client.0.smithi170.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T17:33:34.603 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.603 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.603 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.604 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T17:33:34.604 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.604 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.604 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T17:33:34.604 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.604 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/92084/exe 2023-10-30T17:33:34.605 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.605 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.605 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.605 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.605 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.606 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.606 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.606 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.606 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T17:33:34.606 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.607 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T17:33:34.607 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T17:33:34.607 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.607 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.607 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.608 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.608 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.608 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T17:33:34.608 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.608 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T17:33:34.609 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.609 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.609 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.609 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.609 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T17:33:34.609 INFO:tasks.workunit.client.0.smithi170.stderr:+++ return 2023-10-30T17:33:34.610 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.610 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.610 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T17:33:34.610 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.610 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T17:33:34.611 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.611 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export S_COLORS=auto 2023-10-30T17:33:34.611 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.611 INFO:tasks.workunit.client.0.smithi170.stderr:+++ S_COLORS=auto 2023-10-30T17:33:34.611 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.612 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T17:33:34.612 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.612 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T17:33:34.612 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.612 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_api_cls_remote_reads 2023-10-30T17:33:34.612 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-10-30T17:33:34.613 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-10-30T17:33:34.613 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.613 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.613 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.613 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.614 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.614 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.614 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.614 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.614 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.615 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.615 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.615 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.615 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.615 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.616 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.616 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.616 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.616 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.616 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.616 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.617 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.617 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.617 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.617 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.617 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.618 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.618 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.618 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.618 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.618 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.619 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.619 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.619 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.619 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.619 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.620 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.620 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.620 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.620 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.620 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.620 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.621 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.621 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.621 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.621 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.621 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.622 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.622 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.622 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.622 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.622 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.623 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.623 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.623 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.623 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.623 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.624 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.624 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.624 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.624 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.624 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.625 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.625 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.625 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.625 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.625 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.625 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.626 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.626 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.626 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.626 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.626 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.627 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.627 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.627 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.627 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.627 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.628 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.628 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.628 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.628 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.628 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.629 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.629 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/92177/exe 2023-10-30T17:33:34.629 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.629 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.629 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.630 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.630 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.630 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.630 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.630 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/92101/exe 2023-10-30T17:33:34.631 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.631 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.631 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.631 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.631 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.631 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.632 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.632 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.632 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.632 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.632 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.633 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.633 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_open_pools_parallel 2023-10-30T17:33:34.633 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.633 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.633 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.633 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.634 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.634 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.634 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.634 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.634 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.635 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.635 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-10-30T17:33:34.635 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_list_parallel 2023-10-30T17:33:34.635 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-10-30T17:33:34.635 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_list_parallel.log 2023-10-30T17:33:34.636 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ list_parallel: /' 2023-10-30T17:33:34.636 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T17:33:34.636 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T17:33:34.636 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T17:33:34.636 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.637 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T17:33:34.637 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.637 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T17:33:34.637 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T17:33:34.637 INFO:tasks.workunit.client.0.smithi170.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T17:33:34.638 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T17:33:34.638 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T17:33:34.638 INFO:tasks.workunit.client.0.smithi170.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T17:33:34.638 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.638 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T17:33:34.639 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.639 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T17:33:34.639 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T17:33:34.639 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.639 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T17:33:34.640 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.640 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T17:33:34.640 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.640 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T17:33:34.640 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.640 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T17:33:34.641 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup 2023-10-30T17:33:34.641 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/locale 2023-10-30T17:33:34.641 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.641 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T17:33:34.641 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.642 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T17:33:34.642 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T17:33:34.642 INFO:tasks.workunit.client.0.smithi170.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T17:33:34.642 INFO:tasks.workunit.client.0.smithi170.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T17:33:34.642 INFO:tasks.workunit.client.0.smithi170.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T17:33:34.643 INFO:tasks.workunit.client.0.smithi170.stderr:++++ LANG=en_US.UTF-8 2023-10-30T17:33:34.643 INFO:tasks.workunit.client.0.smithi170.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T17:33:34.643 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T17:33:34.643 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.643 INFO:tasks.workunit.client.0.smithi170.stderr:+++ LANG=en_US.UTF-8 2023-10-30T17:33:34.644 INFO:tasks.workunit.client.0.smithi170.stderr:+++ unset LANG_backup config 2023-10-30T17:33:34.644 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n '' ']' 2023-10-30T17:33:34.644 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T17:33:34.644 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' dumb = linux ']' 2023-10-30T17:33:34.644 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.644 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T17:33:34.645 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.645 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/less.sh 2023-10-30T17:33:34.645 INFO:tasks.workunit.client.0.smithi170.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T17:33:34.645 INFO:tasks.workunit.client.0.smithi170.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T17:33:34.645 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T17:33:34.646 INFO:tasks.workunit.client.0.smithi170.stderr:++ '[' '' ']' 2023-10-30T17:33:34.646 INFO:tasks.workunit.client.0.smithi170.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T17:33:34.646 INFO:tasks.workunit.client.0.smithi170.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T17:33:34.646 INFO:tasks.workunit.client.0.smithi170.stderr:+++++ readlink /proc/92193/exe 2023-10-30T17:33:34.646 INFO:tasks.workunit.client.0.smithi170.stderr:++++ basename /usr/bin/bash 2023-10-30T17:33:34.647 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_declare='declare -f' 2023-10-30T17:33:34.647 INFO:tasks.workunit.client.0.smithi170.stderr:+++ which_opt=-f 2023-10-30T17:33:34.647 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export which_declare 2023-10-30T17:33:34.647 INFO:tasks.workunit.client.0.smithi170.stderr:+++ export -f which 2023-10-30T17:33:34.647 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset i 2023-10-30T17:33:34.648 INFO:tasks.workunit.client.0.smithi170.stderr:++ unset -f pathmunge 2023-10-30T17:33:34.648 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T17:33:34.648 INFO:tasks.workunit.client.0.smithi170.stderr:+ export PATH 2023-10-30T17:33:34.648 INFO:tasks.workunit.client.0.smithi170.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T17:33:34.648 INFO:tasks.workunit.client.0.smithi170.stderr:+ unset rc 2023-10-30T17:33:34.649 INFO:tasks.workunit.client.0.smithi170.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-10-30T17:33:34.649 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph_test_rados_delete_pools_parallel 2023-10-30T17:33:34.649 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-10-30T17:33:34.988 INFO:tasks.workunit.client.0.smithi170.stdout:aac96705b0 2023-10-30T17:33:34.988 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.974+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v24) v1 ==== 43+0+0 (secure 0 0 0) 0x7fa64c057fc0 con 0x55aac96705b0 2023-10-30T17:33:34.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.974+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 pool_op_reply(tid 1 (0) Success v24) v1 2023-10-30T17:33:34.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.974+0000 7fa63ffff640 10 client.4414.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v24) v1 2023-10-30T17:33:34.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.974+0000 7fa63ffff640 10 client.4414.objecter have request 1 at 0x55aac9639260 Op: create 2023-10-30T17:33:34.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.974+0000 7fa63ffff640 20 client.4414.objecter waiting for client to reach epoch 24 before calling back 2023-10-30T17:33:34.990 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.974+0000 7fa63ffff640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:33:34.990 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.974+0000 7fa63ffff640 10 monclient: _renew_subs 2023-10-30T17:33:34.990 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.974+0000 7fa63ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:33:34.990 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.974+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=24}) v3 -- 0x7fa644070780 con 0x55aac96705b0 2023-10-30T17:33:34.990 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.974+0000 7fa63ffff640 10 client.4414.objecter done 2023-10-30T17:33:34.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 8 ==== osd_map(24..24 src has 1..24) v4 ==== 2486+0+0 (secure 0 0 0) 0x7fa64c05c5d0 con 0x55aac96705b0 2023-10-30T17:33:34.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(24..24 src has 1..24) v4 2023-10-30T17:33:34.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [24,24] > 23 2023-10-30T17:33:34.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 24 2023-10-30T17:33:34.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:33:34.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 10 librados: set snap read head -> 1 2023-10-30T17:33:34.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 10 librados: writefull oid=exist nspace= 2023-10-30T17:33:34.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac9634eb0 2023-10-30T17:33:34.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:34.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:34.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:33:34.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x55aac9639a80 0x55aac9635860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:33:34.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:34.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 2 osd.3 2023-10-30T17:33:34.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:34.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 2 2023-10-30T17:33:34.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 15 client.4414.objecter _send_op 2 to 5.1 on osd.3 2023-10-30T17:33:36.695 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:02:20.661 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:36.753 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:02:20.719 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:36.757 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:02:20.722 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:36.768 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:20.707 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:36.773 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:20.712 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:36.788 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:02:20.754 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:36.813 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:20.752 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:36.840 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:02:20.805 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:36.884 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:02:20.850 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:39.821 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:2 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) v8 -- 0x55aac9635da0 con 0x55aac9639a80 2023-10-30T17:33:39.822 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:39.822 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:33:39.822 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.987+0000 7fa6570cc640 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x55aac9639a80 0x55aac9635860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:33:39.823 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:34.992+0000 7fa6570cc640 10 monclient: get_auth_request con 0x55aac9639a80 auth_method 0 2023-10-30T17:33:39.823 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:35.124+0000 7fa63ffff640 10 client.4414.objecter ms_handle_connect 0x55aac9639a80 2023-10-30T17:33:39.823 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:35.124+0000 7fa6570cc640 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x55aac9639a80 0x55aac9635860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:33:39.823 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:36.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:39.823 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:36.818+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:39.824 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:36.818+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:39.824 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:36.818+0000 7fa63effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:33:39.824 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:37.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:39.824 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:37.818+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:39.824 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:37.818+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:39.825 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:38.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:39.825 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:38.818+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:39.825 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:38.818+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:39.825 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:38.876+0000 7fa658b2f640 10 client.4414.objecter tick 2023-10-30T17:33:39.826 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.595+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 1 ==== osd_map(25..25 src has 1..25) v4 ==== 9053+0+0 (crc 0 0 0) 0x7fa648052440 con 0x55aac9639a80 2023-10-30T17:33:39.826 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.595+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(25..25 src has 1..25) v4 2023-10-30T17:33:39.826 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.595+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [25,25] > 24 2023-10-30T17:33:39.826 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.595+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 25 2023-10-30T17:33:39.826 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.595+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:39.827 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.595+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 25 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:39.827 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.595+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:39.827 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.595+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:33:39.827 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.595+0000 7fa63ffff640 20 client.4414.objecter 2 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2023-10-30T17:33:48.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:48.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.818+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:48.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:39.818+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:48.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:40.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:48.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:40.818+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:48.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:40.818+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:48.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:41.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:48.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:41.818+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:48.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:41.818+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:48.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:42.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:48.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:42.818+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:48.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:42.818+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:48.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:43.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:48.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:43.818+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:48.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:43.818+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:48.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:43.877+0000 7fa658b2f640 10 client.4414.objecter tick 2023-10-30T17:33:48.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:44.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:48.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:44.818+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:48.883 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:44.818+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:48.883 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:45.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:48.883 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:45.818+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:48.883 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:45.818+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:48.883 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:46.818+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:48.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:46.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:48.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:46.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:48.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:46.819+0000 7fa63effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:33:48.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:47.819+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:48.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:47.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:48.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:47.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:48.885 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.819+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:48.885 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:48.885 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:48.885 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.878+0000 7fa658b2f640 10 client.4414.objecter tick 2023-10-30T17:33:48.885 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.878+0000 7fa658b2f640 2 client.4414.objecter tid 2 on osd.3 is laggy 2023-10-30T17:33:48.886 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.878+0000 7fa658b2f640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:33:48.886 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.878+0000 7fa658b2f640 10 monclient: _renew_subs 2023-10-30T17:33:48.886 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.878+0000 7fa658b2f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:33:48.926 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.878+0000 7fa658b2f640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=26}) v3 -- 0x7fa628002f90 con 0x55aac96705b0 2023-10-30T17:33:48.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.878+0000 7fa658b2f640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- ping magic: 0 v1 -- 0x7fa628003320 con 0x55aac9639a80 2023-10-30T17:33:48.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 9 ==== osd_map(26..34 src has 1..34) v4 ==== 13674+0+0 (secure 0 0 0) 0x7fa64c0565c0 con 0x55aac96705b0 2023-10-30T17:33:48.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(26..34 src has 1..34) v4 2023-10-30T17:33:48.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [26,34] > 25 2023-10-30T17:33:48.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 26 2023-10-30T17:33:48.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:48.928 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 26 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:48.928 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:48.928 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 27 2023-10-30T17:33:48.928 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:48.928 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 27 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:48.929 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:48.929 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 28 2023-10-30T17:33:48.929 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:48.929 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 28 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:48.929 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:48.930 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 29 2023-10-30T17:33:48.930 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:48.930 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 29 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:48.930 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:48.930 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 30 2023-10-30T17:33:48.930 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:48.931 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 30 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:48.931 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:48.931 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 31 2023-10-30T17:33:48.931 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:48.931 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 31 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:53.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:53.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 32 2023-10-30T17:33:53.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:53.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 32 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:53.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:53.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 33 2023-10-30T17:33:53.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:53.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 33 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:53.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:53.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 34 2023-10-30T17:33:53.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:53.883 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 34 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:53.883 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:53.883 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:33:53.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:48.925+0000 7fa63ffff640 20 client.4414.objecter 2 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2023-10-30T17:33:53.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:49.819+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:53.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:49.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:53.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:49.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:53.884 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:50.819+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:53.885 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:50.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:53.885 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:50.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:53.885 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:51.819+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:53.886 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:51.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:53.886 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:51.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:53.886 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:52.819+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:53.887 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:52.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:53.887 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:52.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:53.888 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.819+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:53.888 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:53.888 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:53.889 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.879+0000 7fa658b2f640 10 client.4414.objecter tick 2023-10-30T17:33:53.889 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.879+0000 7fa658b2f640 2 client.4414.objecter tid 2 on osd.3 is laggy 2023-10-30T17:33:53.889 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.879+0000 7fa658b2f640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:33:53.889 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.879+0000 7fa658b2f640 10 monclient: _renew_subs 2023-10-30T17:33:54.671 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.879+0000 7fa658b2f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:33:54.671 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.879+0000 7fa658b2f640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=35}) v3 -- 0x7fa6280034e0 con 0x55aac96705b0 2023-10-30T17:33:54.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.879+0000 7fa658b2f640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- ping magic: 0 v1 -- 0x7fa6280036d0 con 0x55aac9639a80 2023-10-30T17:33:54.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 10 ==== osd_map(35..37 src has 1..37) v4 ==== 2362+0+0 (secure 0 0 0) 0x7fa64c058410 con 0x55aac96705b0 2023-10-30T17:33:54.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(35..37 src has 1..37) v4 2023-10-30T17:33:54.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [35,37] > 34 2023-10-30T17:33:54.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 35 2023-10-30T17:33:54.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:54.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 35 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:54.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:54.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 36 2023-10-30T17:33:54.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:54.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 36 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:54.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:54.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 37 2023-10-30T17:33:54.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 10 client.4414.objecter checking op 2 2023-10-30T17:33:54.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 37 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:54.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:54.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:33:54.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:53.881+0000 7fa63ffff640 20 client.4414.objecter 2 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2023-10-30T17:33:54.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 2 ==== osd_op_reply(2 exist [writefull 0~5] v25'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:33:54.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(2 exist [writefull 0~5] v25'1 uv1 ondisk = 0) v8 2023-10-30T17:33:54.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:33:54.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 2 ondisk uv 1 in 5.1 attempt 0 2023-10-30T17:33:54.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:33:54.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 2 2023-10-30T17:33:54.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa6570cc640 15 client.4414.objecter _finish_op 2 2023-10-30T17:33:54.677 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:54.677 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 2 2023-10-30T17:33:54.677 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:33:54.677 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 10 librados: Objecter returned from writefull r=0 2023-10-30T17:33:54.677 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-10-30T17:33:54.678 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac9634eb0 2023-10-30T17:33:54.678 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 37 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:33:54.678 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:54.678 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:33:54.678 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:54.679 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 3 osd.3 2023-10-30T17:33:54.679 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:54.679 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 3 2023-10-30T17:33:54.679 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 15 client.4414.objecter _send_op 3 to 5.1 on osd.3 2023-10-30T17:33:54.679 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:3 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x55aac96ec3b0 con 0x55aac9639a80 2023-10-30T17:33:54.680 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:54.680 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:33:54.680 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96ec750 2023-10-30T17:33:54.680 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 37 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:33:54.680 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 20 client.4414.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2023-10-30T17:33:54.681 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:33:54.681 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:54.681 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 10 client.4414.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 4 osd.3 2023-10-30T17:33:54.681 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:54.681 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 4 2023-10-30T17:33:54.682 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 15 client.4414.objecter _send_op 4 to 5.1 on osd.3 2023-10-30T17:33:54.682 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:4 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x55aac96ecbf0 con 0x55aac9639a80 2023-10-30T17:33:54.682 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:33:55.497 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.670+0000 7fa659641780 5 client.4414.objecter 2 in flight 2023-10-30T17:33:55.498 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.819+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:55.498 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:55.498 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:54.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:55.498 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:33:55.498 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T17:33:55.499 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:33:55.499 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 3 ondisk uv 1 in 5.1 attempt 0 2023-10-30T17:33:55.499 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 5 2023-10-30T17:33:55.499 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 3 2023-10-30T17:33:55.499 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 15 client.4414.objecter _finish_op 3 2023-10-30T17:33:55.500 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:33:55.500 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 3 2023-10-30T17:33:55.500 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 5 client.4414.objecter 1 in flight 2023-10-30T17:33:55.500 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:33:55.500 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T17:33:55.501 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:33:55.501 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 4 ondisk uv 0 in 5.1 attempt 0 2023-10-30T17:33:55.501 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 10 client.4414.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T17:33:55.501 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:33:55.501 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 4 2023-10-30T17:33:55.502 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 15 client.4414.objecter _finish_op 4 2023-10-30T17:33:55.502 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:55.502 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 4 2023-10-30T17:33:55.502 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:33:55.502 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (826 ms) 2023-10-30T17:33:55.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-10-30T17:33:55.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96ec750 2023-10-30T17:33:55.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 37 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:33:55.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:55.702 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:33:55.702 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:55.703 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 5 osd.3 2023-10-30T17:33:55.703 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:55.703 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 5 2023-10-30T17:33:55.703 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 15 client.4414.objecter _send_op 5 to 5.1 on osd.3 2023-10-30T17:33:55.704 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:5 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x55aac96ecf50 con 0x55aac9639a80 2023-10-30T17:33:55.704 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:55.704 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:33:55.704 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac9634eb0 2023-10-30T17:33:55.704 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 37 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:33:55.705 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 20 client.4414.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2023-10-30T17:33:55.705 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:33:55.705 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:55.705 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 10 client.4414.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 6 osd.3 2023-10-30T17:33:55.705 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:55.706 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 6 2023-10-30T17:33:55.706 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 15 client.4414.objecter _send_op 6 to 5.1 on osd.3 2023-10-30T17:33:55.706 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:6 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x55aac96f16a0 con 0x55aac9639a80 2023-10-30T17:33:55.706 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:33:55.706 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.496+0000 7fa659641780 5 client.4414.objecter 2 in flight 2023-10-30T17:33:55.707 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.700+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:33:55.707 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T17:33:55.707 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:33:55.707 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 5 ondisk uv 1 in 5.1 attempt 0 2023-10-30T17:33:55.707 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 5 2023-10-30T17:33:55.707 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 5 2023-10-30T17:33:55.708 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 15 client.4414.objecter _finish_op 5 2023-10-30T17:33:55.708 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:33:55.708 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 5 2023-10-30T17:33:55.708 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 5 client.4414.objecter 1 in flight 2023-10-30T17:33:55.708 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:33:55.709 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T17:33:55.709 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:33:55.709 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 6 ondisk uv 0 in 5.1 attempt 0 2023-10-30T17:33:55.709 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 10 client.4414.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T17:33:55.709 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:33:55.710 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 6 2023-10-30T17:33:55.710 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 15 client.4414.objecter _finish_op 6 2023-10-30T17:33:55.710 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:55.710 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 6 2023-10-30T17:33:55.710 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:33:55.710 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (205 ms) 2023-10-30T17:33:55.711 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-10-30T17:33:55.711 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac9634eb0 2023-10-30T17:33:55.711 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 37 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:33:55.711 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:55.711 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:33:55.712 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:55.712 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 7 osd.3 2023-10-30T17:33:55.712 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:55.712 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 7 2023-10-30T17:33:55.712 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 15 client.4414.objecter _send_op 7 to 5.1 on osd.3 2023-10-30T17:33:55.712 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:7 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x55aac96ec730 con 0x55aac9639a80 2023-10-30T17:33:55.713 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:55.713 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:33:55.739 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96f2070 2023-10-30T17:33:55.739 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 37 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:33:55.740 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 20 client.4414.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2023-10-30T17:33:55.740 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:33:55.740 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:55.740 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 10 client.4414.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 8 osd.3 2023-10-30T17:33:55.740 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:55.741 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 8 2023-10-30T17:33:55.741 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 15 client.4414.objecter _send_op 8 to 5.1 on osd.3 2023-10-30T17:33:55.741 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:8 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37) v8 -- 0x55aac96f2510 con 0x55aac9639a80 2023-10-30T17:33:55.741 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:33:55.742 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.701+0000 7fa659641780 5 client.4414.objecter 2 in flight 2023-10-30T17:33:55.742 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:33:55.742 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T17:33:55.742 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:33:55.742 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 7 ondisk uv 1 in 5.1 attempt 0 2023-10-30T17:33:55.743 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 5 2023-10-30T17:33:55.743 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 7 2023-10-30T17:33:55.743 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 15 client.4414.objecter _finish_op 7 2023-10-30T17:33:55.743 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:33:55.743 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 7 2023-10-30T17:33:55.744 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 5 client.4414.objecter 1 in flight 2023-10-30T17:33:55.744 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:33:55.744 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T17:33:55.744 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:33:55.744 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 8 ondisk uv 0 in 5.1 attempt 0 2023-10-30T17:33:56.748 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:02:40.714 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:56.751 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:02:40.717 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:56.753 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:02:40.719 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:56.813 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:40.751 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:56.826 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:40.765 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:56.836 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:02:40.802 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:56.972 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:02:40.938 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:56.989 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:40.928 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:56.997 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:02:40.963 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:33:57.671 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 10 client.4414.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T17:33:57.671 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:33:57.671 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 8 2023-10-30T17:33:57.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 15 client.4414.objecter _finish_op 8 2023-10-30T17:33:57.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:57.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 8 2023-10-30T17:33:57.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:33:57.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (37 ms) 2023-10-30T17:33:57.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-10-30T17:33:57.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T17:33:57.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 20 librados: queue_aio_write 0x55aac95e6ea0 completion 0x55aac9634eb0 write_seq 1 2023-10-30T17:33:57.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96f1a00 2023-10-30T17:33:57.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 37 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:57.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:57.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:33:57.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:57.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 9 osd.3 2023-10-30T17:33:57.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:57.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 9 2023-10-30T17:33:57.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 15 client.4414.objecter _send_op 9 to 5.1 on osd.3 2023-10-30T17:33:57.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:9 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) v8 -- 0x55aac9634fc0 con 0x55aac9639a80 2023-10-30T17:33:57.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:57.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:33:57.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.738+0000 7fa659641780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T17:33:57.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.819+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:57.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:57.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:55.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:57.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:56.819+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:33:57.676 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:56.819+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:33:57.677 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:56.819+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:33:57.677 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:56.819+0000 7fa63effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:33:57.677 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (1931 ms) 2023-10-30T17:33:57.677 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-10-30T17:33:57.720 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 9 ==== osd_op_reply(9 exist [write 0~5] v37'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:33:57.720 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(9 exist [write 0~5] v37'2 uv2 ondisk = 0) v8 2023-10-30T17:33:57.720 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:33:57.721 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 9 ondisk uv 2 in 5.1 attempt 0 2023-10-30T17:33:57.721 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:33:57.721 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 9 2023-10-30T17:33:57.721 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 15 client.4414.objecter _finish_op 9 2023-10-30T17:33:57.721 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:57.721 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 9 2023-10-30T17:33:57.722 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:33:57.722 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa6570cc640 20 librados: complete_aio_write 0x55aac9634eb0 2023-10-30T17:33:57.722 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T17:33:57.722 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 20 librados: queue_aio_write 0x55aac95e6ea0 completion 0x55aac96f1ea0 write_seq 2 2023-10-30T17:33:57.723 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96f1a00 2023-10-30T17:33:57.723 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 37 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:33:57.723 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:33:57.723 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:33:57.723 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:57.723 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 10 osd.3 2023-10-30T17:33:57.724 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:33:57.724 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 10 2023-10-30T17:33:57.724 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 15 client.4414.objecter _send_op 10 to 5.1 on osd.3 2023-10-30T17:33:57.724 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:10 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) v8 -- 0x55aac9634fc0 con 0x55aac9639a80 2023-10-30T17:33:57.724 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:33:57.724 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:33:57.725 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.669+0000 7fa659641780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T17:33:57.725 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 10 ==== osd_map(38..39 src has 1..40) v4 ==== 1978+0+0 (crc 0 0 0) 0x7fa64805ed90 con 0x55aac9639a80 2023-10-30T17:33:57.725 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(38..39 src has 1..40) v4 2023-10-30T17:34:01.430 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [38,39] > 37 2023-10-30T17:34:01.431 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 38 2023-10-30T17:34:01.431 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 10 client.4414.objecter checking op 10 2023-10-30T17:34:01.431 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 38 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:01.431 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:01.432 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 39 2023-10-30T17:34:01.432 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 10 client.4414.objecter checking op 10 2023-10-30T17:34:01.432 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 39 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:01.432 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:01.432 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:34:01.433 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.718+0000 7fa63ffff640 20 client.4414.objecter 10 5.d3a59201 osd.3 exist [write 0~5 in=5b] 2023-10-30T17:34:01.433 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:01.433 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:01.433 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:57.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:01.433 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:58.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:01.434 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:58.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:01.434 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:58.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:01.434 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:58.880+0000 7fa658b2f640 10 client.4414.objecter tick 2023-10-30T17:34:01.434 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:59.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:01.434 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:59.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:01.435 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:33:59.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:01.435 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:00.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:01.435 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:00.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:01.435 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:00.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:01.435 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 11 ==== osd_op_reply(10 exist [write 0~5] v39'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:01.436 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(10 exist [write 0~5] v39'3 uv3 ondisk = 0) v8 2023-10-30T17:34:01.436 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:01.436 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 10 ondisk uv 3 in 5.1 attempt 0 2023-10-30T17:34:01.436 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:34:01.436 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 10 2023-10-30T17:34:01.437 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 15 client.4414.objecter _finish_op 10 2023-10-30T17:34:01.437 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.438 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 10 2023-10-30T17:34:02.438 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:34:02.438 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa6570cc640 20 librados: complete_aio_write 0x55aac96f1ea0 2023-10-30T17:34:02.439 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3760 ms) 2023-10-30T17:34:02.439 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-10-30T17:34:02.439 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T17:34:02.439 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 20 librados: queue_aio_write 0x55aac95e6ea0 completion 0x55aac96f2370 write_seq 3 2023-10-30T17:34:02.439 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96f1a00 2023-10-30T17:34:02.440 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 39 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:02.440 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:02.440 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:34:02.440 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:02.440 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 11 osd.3 2023-10-30T17:34:02.441 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:02.441 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 11 2023-10-30T17:34:02.441 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 15 client.4414.objecter _send_op 11 to 5.1 on osd.3 2023-10-30T17:34:02.441 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:11 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) v8 -- 0x55aac9634fc0 con 0x55aac9639a80 2023-10-30T17:34:02.441 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.442 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:34:02.442 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.429+0000 7fa659641780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T17:34:02.442 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:02.442 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:02.442 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:01.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:02.443 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 12 ==== osd_map(40..43 src has 1..43) v4 ==== 2993+0+0 (crc 0 0 0) 0x7fa648052880 con 0x55aac9639a80 2023-10-30T17:34:02.443 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 13 ==== osd_op_reply(11 exist [write 0~5] v43'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:02.443 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(40..43 src has 1..43) v4 2023-10-30T17:34:02.445 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(11 exist [write 0~5] v43'4 uv4 ondisk = 0) v8 2023-10-30T17:34:02.445 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:02.445 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [40,43] > 39 2023-10-30T17:34:02.445 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 40 2023-10-30T17:34:02.446 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 10 client.4414.objecter checking op 11 2023-10-30T17:34:02.446 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 40 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:02.446 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:02.446 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 41 2023-10-30T17:34:02.446 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 10 client.4414.objecter checking op 11 2023-10-30T17:34:02.447 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 41 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:02.447 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:02.447 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 42 2023-10-30T17:34:02.447 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 10 client.4414.objecter checking op 11 2023-10-30T17:34:02.447 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 42 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:02.448 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:02.448 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 43 2023-10-30T17:34:02.448 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 10 client.4414.objecter checking op 11 2023-10-30T17:34:02.448 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 43 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:02.448 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:02.448 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:34:02.449 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa63ffff640 20 client.4414.objecter 11 5.d3a59201 osd.3 exist [write 0~5 in=5b] 2023-10-30T17:34:02.449 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 11 ondisk uv 4 in 5.1 attempt 0 2023-10-30T17:34:02.449 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:34:02.449 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 11 2023-10-30T17:34:02.449 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 15 client.4414.objecter _finish_op 11 2023-10-30T17:34:02.450 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.450 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 11 2023-10-30T17:34:02.450 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:34:02.450 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa6570cc640 20 librados: complete_aio_write 0x55aac96f2370 2023-10-30T17:34:02.450 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1007 ms) 2023-10-30T17:34:02.451 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-10-30T17:34:02.451 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96f1a00 2023-10-30T17:34:02.451 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 43 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:34:02.451 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:02.518 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] api_service: [==========] Running 4 tests from 1 test suite. 2023-10-30T17:34:02.518 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [----------] Global test environment set-up. 2023-10-30T17:34:02.518 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [----------] 4 tests from LibRadosService 2023-10-30T17:34:02.519 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-10-30T17:34:02.519 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5271 ms) 2023-10-30T17:34:02.519 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-10-30T17:34:02.519 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [ OK ] LibRadosService.RegisterLate (71 ms) 2023-10-30T17:34:02.519 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-10-30T17:34:02.520 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: cluster: 2023-10-30T17:34:02.520 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: id: dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:34:02.520 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: health: HEALTH_WARN 2023-10-30T17:34:02.520 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 19 pool(s) do not have an application enabled 2023-10-30T17:34:02.520 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 2023-10-30T17:34:02.520 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: services: 2023-10-30T17:34:02.521 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-10-30T17:34:02.521 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: mgr: x(active, since 115s), standbys: y 2023-10-30T17:34:02.521 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: osd: 8 osds: 8 up (since 72s), 8 in (since 116s) 2023-10-30T17:34:02.521 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-10-30T17:34:02.521 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 2023-10-30T17:34:02.522 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: data: 2023-10-30T17:34:02.522 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: pools: 26 pools, 777 pgs 2023-10-30T17:34:02.522 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: objects: 2 objects, 449 KiB 2023-10-30T17:34:02.522 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-10-30T17:34:02.522 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: pgs: 86.358% pgs unknown 2023-10-30T17:34:02.523 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 12.484% pgs not active 2023-10-30T17:34:02.523 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 671 unknown 2023-10-30T17:34:02.523 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 97 creating+peering 2023-10-30T17:34:02.523 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 9 active+clean 2023-10-30T17:34:02.523 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 2023-10-30T17:34:02.524 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 2023-10-30T17:34:02.524 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: cluster: 2023-10-30T17:34:02.524 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: id: dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:34:02.524 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: health: HEALTH_WARN 2023-10-30T17:34:02.524 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 7 pool(s) do not have an application enabled 2023-10-30T17:34:02.524 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 2023-10-30T17:34:02.525 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: services: 2023-10-30T17:34:02.525 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-10-30T17:34:02.525 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: mgr: x(active, since 117s), standbys: y 2023-10-30T17:34:02.525 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: osd: 8 osds: 8 up (since 74s), 8 in (since 118s) 2023-10-30T17:34:02.525 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-10-30T17:34:02.525 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-10-30T17:34:02.526 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 2023-10-30T17:34:02.526 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: data: 2023-10-30T17:34:02.526 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: pools: 28 pools, 817 pgs 2023-10-30T17:34:02.526 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: objects: 2 objects, 449 KiB 2023-10-30T17:34:02.526 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-10-30T17:34:02.527 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: pgs: 54.100% pgs unknown 2023-10-30T17:34:02.527 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 43.696% pgs not active 2023-10-30T17:34:02.527 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 442 unknown 2023-10-30T17:34:02.527 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 310 creating+peering 2023-10-30T17:34:02.527 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 47 creating+activating 2023-10-30T17:34:02.528 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 18 active+clean 2023-10-30T17:34:02.528 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 2023-10-30T17:34:02.528 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 2023-10-30T17:34:02.528 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [ OK ] LibRadosService.StatusFormat (3152 ms) 2023-10-30T17:34:02.528 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [ RUN ] LibRadosService.Status 2023-10-30T17:34:02.529 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [ OK ] LibRadosService.Status (20078 ms) 2023-10-30T17:34:02.529 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [----------] 4 tests from LibRadosService (28572 ms total) 2023-10-30T17:34:02.529 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: 2023-10-30T17:34:02.529 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [----------] Global test environment tear-down 2023-10-30T17:34:02.529 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [==========] 4 tests from 1 test suite ran. (28572 ms total) 2023-10-30T17:34:02.530 INFO:tasks.workunit.client.0.smithi170.stdout: api_service: [ PASSED ] 4 tests. 2023-10-30T17:34:02.655 INFO:tasks.workunit.client.0.smithi170.stdout:primary 3 2023-10-30T17:34:02.655 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:02.655 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 12 osd.3 2023-10-30T17:34:02.656 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:02.656 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 12 2023-10-30T17:34:02.656 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 15 client.4414.objecter _send_op 12 to 5.1 on osd.3 2023-10-30T17:34:02.656 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:12 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) v8 -- 0x55aac9634fc0 con 0x55aac9639a80 2023-10-30T17:34:02.657 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.657 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:34:02.657 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac97b9740 2023-10-30T17:34:02.658 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 43 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:34:02.658 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 20 client.4414.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2023-10-30T17:34:02.659 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:34:02.659 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.659 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 10 client.4414.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 13 osd.3 2023-10-30T17:34:02.660 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.660 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 13 2023-10-30T17:34:02.660 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 15 client.4414.objecter _send_op 13 to 5.1 on osd.3 2023-10-30T17:34:02.661 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:13 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) v8 -- 0x55aac97b9be0 con 0x55aac9639a80 2023-10-30T17:34:02.661 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:34:02.662 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.436+0000 7fa659641780 5 client.4414.objecter 2 in flight 2023-10-30T17:34:02.662 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 14 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:02.662 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T17:34:02.663 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:02.663 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 12 ondisk uv 4 in 5.1 attempt 0 2023-10-30T17:34:02.663 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 5 2023-10-30T17:34:02.664 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 12 2023-10-30T17:34:02.664 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 15 client.4414.objecter _finish_op 12 2023-10-30T17:34:02.665 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:34:02.665 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 12 2023-10-30T17:34:02.665 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 5 client.4414.objecter 1 in flight 2023-10-30T17:34:02.666 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 15 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:02.666 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T17:34:02.667 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:02.667 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 13 ondisk uv 0 in 5.1 attempt 0 2023-10-30T17:34:02.667 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 10 client.4414.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T17:34:02.668 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:34:02.668 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 13 2023-10-30T17:34:02.669 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 15 client.4414.objecter _finish_op 13 2023-10-30T17:34:02.669 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.669 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 13 2023-10-30T17:34:02.670 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:34:02.670 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (217 ms) 2023-10-30T17:34:02.670 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-10-30T17:34:02.671 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96f1a00 2023-10-30T17:34:02.671 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 43 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:34:02.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:02.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:34:02.672 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:02.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 14 osd.3 2023-10-30T17:34:02.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:02.673 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 14 2023-10-30T17:34:02.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 15 client.4414.objecter _send_op 14 to 5.1 on osd.3 2023-10-30T17:34:02.674 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:14 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) v8 -- 0x55aac97b9740 con 0x55aac9639a80 2023-10-30T17:34:02.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.675 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:34:02.871 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac97b9be0 2023-10-30T17:34:02.872 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 43 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:34:02.872 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 20 client.4414.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2023-10-30T17:34:02.872 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:34:02.872 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.872 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 10 client.4414.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 15 osd.3 2023-10-30T17:34:02.873 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.873 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 15 2023-10-30T17:34:02.873 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 15 client.4414.objecter _send_op 15 to 5.1 on osd.3 2023-10-30T17:34:02.873 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:15 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) v8 -- 0x55aac97ba080 con 0x55aac9639a80 2023-10-30T17:34:02.873 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:34:02.874 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.653+0000 7fa659641780 5 client.4414.objecter 2 in flight 2023-10-30T17:34:02.874 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:02.874 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:02.874 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:02.874 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 16 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:02.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T17:34:02.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:02.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 14 ondisk uv 4 in 5.1 attempt 0 2023-10-30T17:34:02.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 5 2023-10-30T17:34:02.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 14 2023-10-30T17:34:02.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 15 client.4414.objecter _finish_op 14 2023-10-30T17:34:02.876 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:34:02.876 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 14 2023-10-30T17:34:02.876 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 5 client.4414.objecter 1 in flight 2023-10-30T17:34:02.876 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 17 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:02.876 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T17:34:02.876 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:02.877 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 15 ondisk uv 0 in 5.1 attempt 0 2023-10-30T17:34:02.877 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 10 client.4414.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T17:34:02.877 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:34:02.877 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 15 2023-10-30T17:34:02.877 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 15 client.4414.objecter _finish_op 15 2023-10-30T17:34:02.878 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.878 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 15 2023-10-30T17:34:02.878 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:34:02.878 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (217 ms) 2023-10-30T17:34:02.878 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-10-30T17:34:02.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96f1a00 2023-10-30T17:34:02.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 43 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:34:02.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:02.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:34:02.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:02.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 16 osd.3 2023-10-30T17:34:02.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:02.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 16 2023-10-30T17:34:02.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 15 client.4414.objecter _send_op 16 to 5.1 on osd.3 2023-10-30T17:34:02.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:16 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) v8 -- 0x55aac97b9740 con 0x55aac9639a80 2023-10-30T17:34:02.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:34:02.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac97db730 2023-10-30T17:34:02.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 43 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T17:34:02.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 20 client.4414.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2023-10-30T17:34:02.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:34:02.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:02.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 10 client.4414.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 17 osd.3 2023-10-30T17:34:02.882 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:03.125 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 17 2023-10-30T17:34:03.125 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 15 client.4414.objecter _send_op 17 to 5.1 on osd.3 2023-10-30T17:34:03.125 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:17 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) v8 -- 0x55aac97ba250 con 0x55aac9639a80 2023-10-30T17:34:03.125 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:34:03.126 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:02.870+0000 7fa659641780 5 client.4414.objecter 2 in flight 2023-10-30T17:34:03.126 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (252 ms) 2023-10-30T17:34:03.126 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-10-30T17:34:03.126 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 18 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:03.126 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T17:34:03.127 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:03.127 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 16 ondisk uv 4 in 5.1 attempt 0 2023-10-30T17:34:03.127 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 5 2023-10-30T17:34:03.127 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 16 2023-10-30T17:34:03.127 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 15 client.4414.objecter _finish_op 16 2023-10-30T17:34:03.128 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 5 2023-10-30T17:34:03.128 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 16 2023-10-30T17:34:03.128 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 5 client.4414.objecter 1 in flight 2023-10-30T17:34:03.128 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 20 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:03.128 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T17:34:03.129 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:03.129 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 17 ondisk uv 0 in 5.1 attempt 0 2023-10-30T17:34:03.129 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 10 client.4414.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T17:34:03.129 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:34:03.129 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 17 2023-10-30T17:34:03.130 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 15 client.4414.objecter _finish_op 17 2023-10-30T17:34:03.130 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:03.130 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 17 2023-10-30T17:34:03.130 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:34:03.134 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 19 ==== osd_map(44..44 src has 1..44) v4 ==== 1682+0+0 (crc 0 0 0) 0x7fa648052a80 con 0x55aac9639a80 2023-10-30T17:34:03.134 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(44..44 src has 1..44) v4 2023-10-30T17:34:03.134 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [44,44] > 43 2023-10-30T17:34:03.134 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 44 2023-10-30T17:34:03.134 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa63ffff640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:34:03.135 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa63ffff640 10 monclient: _renew_subs 2023-10-30T17:34:03.135 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa63ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:34:03.135 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.122+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=45}) v3 -- 0x7fa64408a470 con 0x55aac96705b0 2023-10-30T17:34:03.135 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:34:03.135 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 20 librados: queue_aio_write 0x55aac95e6ea0 completion 0x55aac96351e0 write_seq 4 2023-10-30T17:34:03.136 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96f1a00 2023-10-30T17:34:03.136 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 44 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:03.136 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:03.136 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:34:03.136 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:03.137 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 18 osd.3 2023-10-30T17:34:03.137 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:03.137 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 18 2023-10-30T17:34:03.137 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 15 client.4414.objecter _send_op 18 to 5.1 on osd.3 2023-10-30T17:34:03.137 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:18 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) v8 -- 0x55aac97b9740 con 0x55aac9639a80 2023-10-30T17:34:03.138 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:03.138 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.123+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:34:03.138 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 11 ==== osd_map(45..45 src has 1..45) v4 ==== 696+0+0 (secure 0 0 0) 0x7fa64c054b90 con 0x55aac96705b0 2023-10-30T17:34:03.138 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(45..45 src has 1..45) v4 2023-10-30T17:34:03.138 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [45,45] > 44 2023-10-30T17:34:03.139 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 45 2023-10-30T17:34:03.139 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 10 client.4414.objecter checking op 18 2023-10-30T17:34:04.148 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 20 client.4414.objecter _ api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-10-30T17:34:04.148 INFO:tasks.workunit.client.0.smithi170.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-10-30T17:34:04.149 INFO:tasks.workunit.client.0.smithi170.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-10-30T17:34:04.149 INFO:tasks.workunit.client.0.smithi170.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-10-30T17:34:04.149 INFO:tasks.workunit.client.0.smithi170.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (30127 ms) 2023-10-30T17:34:04.149 INFO:tasks.workunit.client.0.smithi170.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (30127 ms total) 2023-10-30T17:34:04.149 INFO:tasks.workunit.client.0.smithi170.stdout: api_cls_remote_reads: 2023-10-30T17:34:04.150 INFO:tasks.workunit.client.0.smithi170.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-10-30T17:34:04.150 INFO:tasks.workunit.client.0.smithi170.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (30127 ms total) 2023-10-30T17:34:04.150 INFO:tasks.workunit.client.0.smithi170.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-10-30T17:34:04.923 INFO:tasks.workunit.client.0.smithi170.stdout:calc_target epoch 45 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:04.923 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:04.924 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:34:04.924 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 10 monclient: _renew_subs 2023-10-30T17:34:04.924 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:34:04.924 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=46}) v3 -- 0x7fa644076bb0 con 0x55aac96705b0 2023-10-30T17:34:04.925 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:34:04.925 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.132+0000 7fa63ffff640 20 client.4414.objecter 18 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2023-10-30T17:34:04.925 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:04.925 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:04.925 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:04.926 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:03.881+0000 7fa658b2f640 10 client.4414.objecter tick 2023-10-30T17:34:04.926 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 12 ==== osd_map(46..46 src has 1..46) v4 ==== 1067+0+0 (secure 0 0 0) 0x7fa64c054b90 con 0x55aac96705b0 2023-10-30T17:34:04.926 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(46..46 src has 1..46) v4 2023-10-30T17:34:04.926 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [46,46] > 45 2023-10-30T17:34:04.926 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 46 2023-10-30T17:34:04.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 10 client.4414.objecter checking op 18 2023-10-30T17:34:04.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 46 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:04.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:04.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:34:04.927 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 10 monclient: _renew_subs 2023-10-30T17:34:04.928 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:34:04.928 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=47}) v3 -- 0x7fa644076bb0 con 0x55aac96705b0 2023-10-30T17:34:04.928 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:34:04.928 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.205+0000 7fa63ffff640 20 client.4414.objecter 18 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2023-10-30T17:34:04.928 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:04.929 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:04.929 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:04.929 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 21 ==== osd_op_reply(18 exist [writefull 0~5] v44'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:05.286 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570 api_misc: [==========] Running 12 tests from 4 test suites. 2023-10-30T17:34:05.286 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [----------] Global test environment set-up. 2023-10-30T17:34:05.287 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-10-30T17:34:05.287 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-10-30T17:34:05.287 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-10-30T17:34:05.287 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-10-30T17:34:05.287 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: 2023-10-30T17:34:05.287 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-10-30T17:34:05.288 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-10-30T17:34:05.288 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2023-10-30T17:34:05.288 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: 2023-10-30T17:33:34.028+0000 7f2158d7f780 -1 failed for service _ceph-mon._tcp 2023-10-30T17:34:05.288 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: 2023-10-30T17:33:34.028+0000 7f2158d7f780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-10-30T17:34:05.288 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (247 ms) 2023-10-30T17:34:05.289 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-10-30T17:34:05.289 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2009 ms) 2023-10-30T17:34:05.289 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2256 ms total) 2023-10-30T17:34:05.289 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: 2023-10-30T17:34:05.289 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-10-30T17:34:05.290 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-10-30T17:34:05.290 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2af5eca0 2023-10-30T17:34:05.290 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb6dcf0 2023-10-30T17:34:05.290 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb6f520 2023-10-30T17:34:05.291 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2b216a10 2023-10-30T17:34:05.291 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2b2124d0 2023-10-30T17:34:05.291 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bc38210 2023-10-30T17:34:05.291 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bc38c30 2023-10-30T17:34:05.291 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bc39630 2023-10-30T17:34:05.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bc3a120 2023-10-30T17:34:05.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bc3abb0 2023-10-30T17:34:05.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bc39e50 2023-10-30T17:34:05.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bc3b730 2023-10-30T17:34:05.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb97b20 2023-10-30T17:34:05.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb98560 2023-10-30T17:34:05.293 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bc3a930 2023-10-30T17:34:05.293 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb993e0 2023-10-30T17:34:05.293 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb98370 2023-10-30T17:34:05.293 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb9a2a0 2023-10-30T17:34:05.294 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb99190 2023-10-30T17:34:05.294 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb9ad90 2023-10-30T17:34:05.294 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb9b7d0 2023-10-30T17:34:05.294 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb9a090 2023-10-30T17:34:05.294 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb9c690 2023-10-30T17:34:05.295 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 0x55db2bb9b5a0 2023-10-30T17:34:05.295 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: started 24 aios 2023-10-30T17:34:05.295 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2af5eca0 2023-10-30T17:34:05.295 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb6dcf0 2023-10-30T17:34:05.295 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb6f520 2023-10-30T17:34:05.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2b216a10 2023-10-30T17:34:05.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2b2124d0 2023-10-30T17:34:05.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bc38210 2023-10-30T17:34:05.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bc38c30 2023-10-30T17:34:05.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bc39630 2023-10-30T17:34:05.297 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bc3a120 2023-10-30T17:34:05.297 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bc3abb0 2023-10-30T17:34:05.297 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bc39e50 2023-10-30T17:34:05.297 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bc3b730 2023-10-30T17:34:05.297 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb97b20 2023-10-30T17:34:05.297 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb98560 2023-10-30T17:34:05.298 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bc3a930 2023-10-30T17:34:05.298 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb993e0 2023-10-30T17:34:05.298 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb98370 2023-10-30T17:34:05.298 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb9a2a0 2023-10-30T17:34:05.298 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb99190 2023-10-30T17:34:05.299 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb9ad90 2023-10-30T17:34:05.299 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb9b7d0 2023-10-30T17:34:05.299 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb9a090 2023-10-30T17:34:05.299 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb9c690 2023-10-30T17:34:05.299 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: waiting 0x55db2bb9b5a0 2023-10-30T17:34:05.300 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: done. 2023-10-30T17:34:05.300 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (29232 ms) 2023-10-30T17:34:05.329 INFO:tasks.workunit.client.0.smithi170.stdout: api_cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(18 exist [writefull 0~5] v44'5 uv5 ondisk = 0) v8 2023-10-30T17:34:05.329 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:05.330 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 18 ondisk uv 5 in 5.1 attempt 0 2023-10-30T17:34:05.330 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:34:05.330 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 18 2023-10-30T17:34:05.330 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570cc640 15 client.4414.objecter _finish_op 18 2023-10-30T17:34:05.330 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:05.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 18 2023-10-30T17:34:05.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:34:05.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa6570cc640 20 librados: complete_aio_write 0x55aac96351e0 2023-10-30T17:34:05.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1800 ms) 2023-10-30T17:34:05.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-10-30T17:34:05.331 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 20 librados: queue_aio_write 0x55aac95e6ea0 completion 0x55aac97dbc20 write_seq 5 2023-10-30T17:34:05.332 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac96f1a00 2023-10-30T17:34:05.332 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 46 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:05.332 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:05.332 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:34:05.332 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:05.333 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 19 osd.3 2023-10-30T17:34:05.333 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:05.333 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 19 2023-10-30T17:34:05.333 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 15 client.4414.objecter _send_op 19 to 5.1 on osd.3 2023-10-30T17:34:05.333 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:19 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) v8 -- 0x55aac97b9740 con 0x55aac9639a80 2023-10-30T17:34:05.334 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:05.334 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:04.922+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:34:05.334 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 13 ==== osd_map(47..47 src has 1..47) v4 ==== 1395+0+0 (secure 0 0 0) 0x7fa64c059d90 con 0x55aac96705b0 2023-10-30T17:34:05.334 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(47..47 src has 1..47) v4 2023-10-30T17:34:05.334 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [47,47] > 46 2023-10-30T17:34:05.335 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 47 2023-10-30T17:34:05.969 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 10 client.4414.objecter checking op 19 2023-10-30T17:34:05.969 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 47 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:05.969 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:05.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:34:05.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 10 monclient: _renew_subs 2023-10-30T17:34:05.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:34:05.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=48}) v3 -- 0x7fa644076bb0 con 0x55aac96705b0 2023-10-30T17:34:05.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:34:05.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.327+0000 7fa63ffff640 20 client.4414.objecter 19 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2023-10-30T17:34:05.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:05.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:05.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:05.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 22 ==== osd_op_reply(19 exist [writefull 0~5] v46'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:05.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(19 exist [writefull 0~5] v46'6 uv6 ondisk = 0) v8 2023-10-30T17:34:05.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:05.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 19 ondisk uv 6 in 5.1 attempt 0 2023-10-30T17:34:05.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:34:05.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 19 2023-10-30T17:34:05.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 15 client.4414.objecter _finish_op 19 2023-10-30T17:34:05.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:05.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 19 2023-10-30T17:34:05.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:34:05.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.967+0000 7fa6570cc640 20 librados: complete_aio_write 0x55aac97dbc20 2023-10-30T17:34:05.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1045 ms) 2023-10-30T17:34:05.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-10-30T17:34:05.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 20 librados: queue_aio_write 0x55aac95e6ea0 completion 0x55aac96351e0 write_seq 6 2023-10-30T17:34:05.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 10 client.4414.objecter _op_submit op 0x55aac97db730 2023-10-30T17:34:05.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 20 client.4414.objecter _calc_target epoch 47 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:05.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:05.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 10 client.4414.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,7] primary 3 2023-10-30T17:34:05.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 20 client.4414.objecter _get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:06.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 10 client.4414.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 20 osd.3 2023-10-30T17:34:06.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 20 client.4414.objecter get_session s=0x55aac9587db0 osd=3 3 2023-10-30T17:34:06.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 15 client.4414.objecter _session_op_assign 3 20 2023-10-30T17:34:06.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 15 client.4414.objecter _send_op 20 to 5.1 on osd.3 2023-10-30T17:34:06.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- osd_op(unknown.0.0:20 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47) v8 -- 0x55aac97b9740 con 0x55aac9639a80 2023-10-30T17:34:06.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:06.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:05.968+0000 7fa659641780 5 client.4414.objecter 1 in flight 2023-10-30T17:34:06.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 14 ==== osd_map(48..48 src has 1..48) v4 ==== 1418+0+0 (secure 0 0 0) 0x7fa64c059da0 con 0x55aac96705b0 2023-10-30T17:34:06.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(48..48 src has 1..48) v4 2023-10-30T17:34:06.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [48,48] > 47 2023-10-30T17:34:06.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 48 2023-10-30T17:34:06.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 10 client.4414.objecter checking op 20 2023-10-30T17:34:06.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 48 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:06.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:06.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:34:06.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 10 monclient: _renew_subs 2023-10-30T17:34:06.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:34:06.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=49}) v3 -- 0x7fa644076bb0 con 0x55aac96705b0 2023-10-30T17:34:06.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:34:06.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.300+0000 7fa63ffff640 20 client.4414.objecter 20 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2023-10-30T17:34:06.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:06.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:06.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:06.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.820+0000 7fa63effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:34:06.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 15 ==== osd_map(49..49 src has 1..49) v4 ==== 700+0+0 (secure 0 0 0) 0x7fa64c0586e0 con 0x55aac96705b0 2023-10-30T17:34:06.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(49..49 src has 1..49) v4 2023-10-30T17:34:06.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [49,49] > 48 2023-10-30T17:34:07.982 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 49 2023-10-30T17:34:07.982 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 10 client.4414.objecter checking op 20 2023-10-30T17:34:07.983 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 49 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:07.983 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:07.983 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:34:07.983 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 10 monclient: _renew_subs 2023-10-30T17:34:07.983 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:34:07.984 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=50}) v3 -- 0x7fa644076bb0 con 0x55aac96705b0 2023-10-30T17:34:07.984 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:34:07.984 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:06.972+0000 7fa63ffff640 20 client.4414.objecter 20 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2023-10-30T17:34:07.984 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.820+0000 7fa63effd640 10 monclient: tick 2023-10-30T17:34:07.985 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.820+0000 7fa63effd640 10 monclient: _check_auth_tickets 2023-10-30T17:34:07.985 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.820+0000 7fa63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:34:07.985 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 <== mon.1 v2:172.21.15.177:3300/0 16 ==== osd_map(50..50 src has 1..50) v4 ==== 336+0+0 (secure 0 0 0) 0x7fa64c0586e0 con 0x55aac96705b0 2023-10-30T17:34:07.986 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_map(50..50 src has 1..50) v4 2023-10-30T17:34:07.986 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map got epochs [50,50] > 49 2023-10-30T17:34:07.986 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 3 client.4414.objecter handle_osd_map decoding incremental epoch 50 2023-10-30T17:34:07.987 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 10 client.4414.objecter checking op 20 2023-10-30T17:34:07.987 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 20 client.4414.objecter _calc_target epoch 50 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T17:34:07.987 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 20 client.4414.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2023-10-30T17:34:07.988 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:34:07.988 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 10 monclient: _renew_subs 2023-10-30T17:34:07.988 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:34:07.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 1 -- 172.21.15.170:0/922528809 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=51}) v3 -- 0x7fa644076bb0 con 0x55aac96705b0 2023-10-30T17:34:07.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 20 client.4414.objecter dump_active .. 0 homeless 2023-10-30T17:34:07.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.977+0000 7fa63ffff640 20 client.4414.objecter 20 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2023-10-30T17:34:07.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.980+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 <== osd.3 v2:172.21.15.170:6800/3657546070 23 ==== osd_op_reply(20 exist [writefull 0~5] v47'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa648052600 con 0x55aac9639a80 2023-10-30T17:34:07.990 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.980+0000 7fa6570cc640 10 client.4414.objecter ms_dispatch 0x55aac95bf3e0 osd_op_reply(20 exist [writefull 0~5] v47'7 uv7 ondisk = 0) v8 2023-10-30T17:34:07.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.980+0000 7fa6570cc640 10 client.4414.objecter in handle_osd_op_reply 2023-10-30T17:34:07.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.980+0000 7fa6570cc640 7 client.4414.objecter handle_osd_op_reply 20 ondisk uv 7 in 5.1 attempt 0 2023-10-30T17:34:07.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.980+0000 7fa6570cc640 10 client.4414.objecter op 0 rval 0 len 0 2023-10-30T17:34:07.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.980+0000 7fa6570cc640 15 client.4414.objecter handle_osd_op_reply completed tid 20 2023-10-30T17:34:07.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.980+0000 7fa6570cc640 15 client.4414.objecter _finish_op 20 2023-10-30T17:34:07.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa6570cc640 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 4 2023-10-30T17:34:07.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa6570cc640 15 client.4414.objecter _session_op_remove 3 20 2023-10-30T17:34:07.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa6570cc640 5 client.4414.objecter 0 in flight 2023-10-30T17:34:07.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa6570cc640 20 librados: complete_aio_write 0x55aac96351e0 2023-10-30T17:34:07.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2013 ms) 2023-10-30T17:34:07.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 10 librados: watch_flush enter 2023-10-30T17:34:07.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 10 librados: watch_flush exit 2023-10-30T17:34:07.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 10 client.4414.objecter close_session for osd.3 2023-10-30T17:34:07.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x55aac9639a80 msgr2=0x55aac9635860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:34:07.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x55aac9639a80 0x55aac9635860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:34:07.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 20 client.4414.objecter put_session s=0x55aac9587db0 osd=3 1 2023-10-30T17:34:07.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 20 client.4414.objecter shutdown clearing up homeless session... 2023-10-30T17:34:07.995 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 10 client.4414.objecter successfully canceled tick 2023-10-30T17:34:07.995 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa64406aa40 msgr2=0x7fa64406cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:34:07.995 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa64406aa40 0x7fa64406cf00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55aac970b6e0 tx=0x7fa65000b1b0 comp rx=0 tx=0).stop 2023-10-30T17:34:07.995 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 10 monclient: shutdown 2023-10-30T17:34:07.995 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:34:07.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aac96705b0 msgr2=0x55aac9639540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:34:07.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 --2- 172.21.15.170:0/922528809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aac96705b0 0x55aac9639540 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa64c013000 tx=0x7fa64c000ed0 comp rx=0 tx=0).stop 2023-10-30T17:34:07.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa6570cc640 1 -- 172.21.15.170:0/922528809 reap_dead start 2023-10-30T17:34:07.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 shutdown_connections 2023-10-30T17:34:07.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 >> 172.21.15.170:0/922528809 conn(0x55aac9677730 msgr2=0x55aac9675820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:34:07.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 shutdown_connections 2023-10-30T17:34:07.997 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 -- 172.21.15.170:0/922528809 wait complete. 2023-10-30T17:34:07.997 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.981+0000 7fa659641780 1 librados: shutdown 2023-10-30T17:34:07.997 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [----------] 12 tests from AsioRados (13310 ms total) 2023-10-30T17:34:07.997 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: 2023-10-30T17:34:07.997 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [----------] Global test environment tear-down 2023-10-30T17:34:07.998 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (34199 ms total) 2023-10-30T17:34:07.998 INFO:tasks.workunit.client.0.smithi170.stdout: api_asio: [ PASSED ] 12 tests. 2023-10-30T17:34:10.607 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-10-30T17:34:10.608 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [----------] Global test environment set-up. 2023-10-30T17:34:10.608 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-10-30T17:34:10.608 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-10-30T17:34:10.608 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5309 ms) 2023-10-30T17:34:10.609 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-10-30T17:34:10.609 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (301 ms) 2023-10-30T17:34:10.609 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-10-30T17:34:10.609 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20476 ms) 2023-10-30T17:34:10.609 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-10-30T17:34:10.610 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: attempt 0 of 20 2023-10-30T17:34:10.610 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: failed to deregister: 2023-10-30T17:34:10.610 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: { 2023-10-30T17:34:10.610 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "epoch": 10, 2023-10-30T17:34:10.610 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "modified": "2023-10-30T17:34:03.007260+0000", 2023-10-30T17:34:10.611 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "services": { 2023-10-30T17:34:10.611 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "laundry": { 2023-10-30T17:34:10.611 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "daemons": { 2023-10-30T17:34:10.611 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "summary": "", 2023-10-30T17:34:10.611 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "close-test-pid92092": { 2023-10-30T17:34:10.611 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "start_epoch": 9, 2023-10-30T17:34:10.612 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "start_stamp": "2023-10-30T17:34:00.218673+0000", 2023-10-30T17:34:10.612 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "gid": 4551, 2023-10-30T17:34:10.612 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "addr": "172.21.15.170:0/2895714515", 2023-10-30T17:34:10.612 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "metadata": { 2023-10-30T17:34:10.612 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "arch": "x86_64", 2023-10-30T17:34:10.613 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "ceph_release": "reef", 2023-10-30T17:34:10.613 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-6865-gfee4af45 (fee4af45ddecf08e014b4503e218408d9e5c5be2) reef (dev)", 2023-10-30T17:34:10.613 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "ceph_version_short": "18.0.0-6865-gfee4af45", 2023-10-30T17:34:10.613 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-10-30T17:34:10.613 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "distro": "centos", 2023-10-30T17:34:10.614 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-10-30T17:34:10.614 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "distro_version": "9", 2023-10-30T17:34:10.614 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "foo": "bar", 2023-10-30T17:34:10.614 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "hostname": "smithi170", 2023-10-30T17:34:10.614 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-10-30T17:34:10.615 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "kernel_version": "5.14.0-378.el9.x86_64", 2023-10-30T17:34:10.615 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "mem_swap_kb": "0", 2023-10-30T17:34:10.615 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "mem_total_kb": "32488736", 2023-10-30T17:34:10.615 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "os": "Linux", 2023-10-30T17:34:10.615 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "this": "that" 2023-10-30T17:34:10.616 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: }, 2023-10-30T17:34:10.616 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: "task_status": {} 2023-10-30T17:34:10.616 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: } 2023-10-30T17:34:10.616 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: } 2023-10-30T17:34:10.616 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: } 2023-10-30T17:34:10.616 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: } 2023-10-30T17:34:10.617 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: } 2023-10-30T17:34:10.617 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: 2023-10-30T17:34:10.617 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: attempt 1 of 20 2023-10-30T17:34:10.617 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10549 ms) 2023-10-30T17:34:10.617 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (36635 ms total) 2023-10-30T17:34:10.618 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: 2023-10-30T17:34:10.618 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [----------] Global test environment tear-down 2023-10-30T17:34:10.618 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (36635 ms total) 2023-10-30T17:34:10.618 INFO:tasks.workunit.client.0.smithi170.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-10-30T17:34:17.357 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:03:01.323 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:17.416 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:03:01.382 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:17.427 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:01.366 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:17.427 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:03:01.393 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:17.438 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:01.377 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:17.499 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:03:01.465 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:17.546 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:01.485 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:17.623 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:03:01.589 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:17.630 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:03:01.596 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:21.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: Running main() from gmock_main.cc 2023-10-30T17:34:21.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-10-30T17:34:21.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [----------] Global test environment set-up. 2023-10-30T17:34:21.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-10-30T17:34:21.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-10-30T17:34:21.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ OK ] LibRadosPools.PoolList (5505 ms) 2023-10-30T17:34:21.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-10-30T17:34:21.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (5481 ms) 2023-10-30T17:34:21.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-10-30T17:34:21.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (4138 ms) 2023-10-30T17:34:21.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-10-30T17:34:21.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4244 ms) 2023-10-30T17:34:21.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-10-30T17:34:21.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3363 ms) 2023-10-30T17:34:21.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-10-30T17:34:21.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6583 ms) 2023-10-30T17:34:21.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-10-30T17:34:21.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4925 ms) 2023-10-30T17:34:21.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-10-30T17:34:21.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5129 ms) 2023-10-30T17:34:21.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-10-30T17:34:21.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8802 ms) 2023-10-30T17:34:21.981 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [----------] 9 tests from LibRadosPools (48170 ms total) 2023-10-30T17:34:21.981 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: 2023-10-30T17:34:21.981 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [----------] Global test environment tear-down 2023-10-30T17:34:21.981 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (48170 ms total) 2023-10-30T17:34:21.981 INFO:tasks.workunit.client.0.smithi170.stdout: api_pool: [ PASSED ] 9 tests. 2023-10-30T17:34:24.965 INFO:tasks.ceph.osd.2.smithi170.stderr:2023-10-30T17:34:24.951+0000 f259640 -1 osd.2 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4455.0:3 20.f 20:f4b50b61:::1.obj:head [write 0~187 in=187b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) 2023-10-30T17:34:26.125 INFO:tasks.ceph.osd.2.smithi170.stderr:2023-10-30T17:34:26.026+0000 f259640 -1 osd.2 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4455.0:3 20.f 20:f4b50b61:::1.obj:head [write 0~187 in=187b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) 2023-10-30T17:34:35.827 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-10-30T17:34:35.828 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-10-30T17:34:35.828 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-10-30T17:34:35.828 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-10-30T17:34:35.828 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: seed 91924 2023-10-30T17:34:35.829 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-10-30T17:34:35.829 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (20470 ms) 2023-10-30T17:34:35.829 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-10-30T17:34:35.829 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (3348 ms) 2023-10-30T17:34:35.829 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-10-30T17:34:35.829 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (200 ms) 2023-10-30T17:34:35.830 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-10-30T17:34:35.830 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (508 ms) 2023-10-30T17:34:35.830 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-10-30T17:34:35.830 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (11294 ms) 2023-10-30T17:34:35.830 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (35820 ms total) 2023-10-30T17:34:35.831 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: 2023-10-30T17:34:35.831 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-10-30T17:34:35.831 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-10-30T17:34:35.831 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (6290 ms) 2023-10-30T17:34:35.831 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-10-30T17:34:35.832 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (2 ms) 2023-10-30T17:34:35.832 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-10-30T17:34:35.832 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1327 ms) 2023-10-30T17:34:35.832 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-10-30T17:34:35.832 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (7476 ms) 2023-10-30T17:34:35.833 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (15096 ms total) 2023-10-30T17:34:35.833 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: 2023-10-30T17:34:35.833 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-10-30T17:34:35.833 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (61950 ms total) 2023-10-30T17:34:35.833 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-10-30T17:34:37.487 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:03:21.453 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:37.526 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:21.465 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:37.531 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:21.470 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:37.540 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:03:21.506 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:37.632 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:21.571 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:37.646 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:03:21.612 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:37.652 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:03:21.618 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:37.669 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:03:21.635 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:37.793 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:03:21.759 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:34:55.836 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-10-30T17:34:55.836 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-10-30T17:34:55.837 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-10-30T17:34:55.837 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-10-30T17:34:55.837 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-10-30T17:34:55.837 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: notify 2023-10-30T17:34:55.837 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (24273 ms) 2023-10-30T17:34:55.838 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-10-30T17:34:55.838 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (7255 ms) 2023-10-30T17:34:55.838 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (31528 ms total) 2023-10-30T17:34:55.838 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: 2023-10-30T17:34:55.838 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-10-30T17:34:55.839 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-10-30T17:34:55.839 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: notify 2023-10-30T17:34:55.839 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (6799 ms) 2023-10-30T17:34:55.839 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-10-30T17:34:55.839 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: notify 2023-10-30T17:34:55.839 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (9662 ms) 2023-10-30T17:34:55.840 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-10-30T17:34:55.840 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (7800 ms) 2023-10-30T17:34:55.840 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-10-30T17:34:55.840 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (6548 ms) 2023-10-30T17:34:55.840 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-10-30T17:34:55.841 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: handle_notify cookie 94034624924160 notify_id 373662154752 notifier_gid 4735 2023-10-30T17:34:55.841 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (4205 ms) 2023-10-30T17:34:55.841 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-10-30T17:34:55.841 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: handle_notify cookie 94034624924160 notify_id 377957122049 notifier_gid 4735 2023-10-30T17:34:55.841 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (1584 ms) 2023-10-30T17:34:55.841 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-10-30T17:34:55.842 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: handle_notify cookie 94034624924160 notify_id 377957122048 notifier_gid 4735 2023-10-30T17:34:55.842 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (648 ms) 2023-10-30T17:34:55.842 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-10-30T17:34:55.842 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: handle_notify cookie 94034626581488 notify_id 377957122049 notifier_gid 4735 2023-10-30T17:34:55.842 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (329 ms) 2023-10-30T17:34:55.843 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-10-30T17:34:55.843 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: handle_notify cookie 94034627360624 notify_id 377957122052 notifier_gid 4735 2023-10-30T17:34:55.843 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (369 ms) 2023-10-30T17:34:55.843 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-10-30T17:34:55.843 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: handle_notify cookie 94034627360624 notify_id 377957122053 notifier_gid 4735 2023-10-30T17:34:55.844 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (234 ms) 2023-10-30T17:34:55.844 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-10-30T17:34:55.844 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: trying... 2023-10-30T17:34:55.844 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: handle_notify cookie 94034627360624 notify_id 377957122049 notifier_gid 4735 2023-10-30T17:34:55.844 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: timed out 2023-10-30T17:34:58.092 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: flus api_cmd_pp: Running main() from gmock_main.cc 2023-10-30T17:34:58.093 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-10-30T17:34:58.093 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-10-30T17:34:58.093 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-10-30T17:34:58.093 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-10-30T17:34:58.094 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (753 ms) 2023-10-30T17:34:58.094 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-10-30T17:34:58.094 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (81 ms) 2023-10-30T17:34:58.094 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-10-30T17:34:58.095 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (83321 ms) 2023-10-30T17:34:58.095 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (84155 ms total) 2023-10-30T17:34:58.095 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: 2023-10-30T17:34:58.095 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-10-30T17:34:58.095 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (84155 ms total) 2023-10-30T17:34:58.096 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-10-30T17:35:02.008 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: Running main() from gmock_main.cc 2023-10-30T17:35:02.008 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-10-30T17:35:02.009 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-10-30T17:35:02.009 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-10-30T17:35:02.009 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-10-30T17:35:02.009 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-10-30T17:35:02.009 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-10-30T17:35:02.009 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (19507 ms) 2023-10-30T17:35:02.010 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-10-30T17:35:02.010 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (3544 ms) 2023-10-30T17:35:02.010 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-10-30T17:35:02.010 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (1607 ms) 2023-10-30T17:35:02.011 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-10-30T17:35:02.011 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (18283 ms) 2023-10-30T17:35:02.011 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-10-30T17:35:02.011 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (3148 ms) 2023-10-30T17:35:02.011 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-10-30T17:35:02.011 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (2414 ms) 2023-10-30T17:35:02.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-10-30T17:35:02.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (918 ms) 2023-10-30T17:35:02.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-10-30T17:35:02.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (3045 ms) 2023-10-30T17:35:02.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-10-30T17:35:02.013 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (2311 ms) 2023-10-30T17:35:02.013 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-10-30T17:35:02.013 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5525 ms) 2023-10-30T17:35:02.013 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-10-30T17:35:02.013 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6083 ms) 2023-10-30T17:35:02.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-10-30T17:35:02.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (4968 ms) 2023-10-30T17:35:02.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-10-30T17:35:02.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (520 ms) 2023-10-30T17:35:02.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-10-30T17:35:02.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (1359 ms) 2023-10-30T17:35:02.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-10-30T17:35:02.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (353 ms) 2023-10-30T17:35:02.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-10-30T17:35:02.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (613 ms) 2023-10-30T17:35:02.016 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (74198 ms total) 2023-10-30T17:35:02.016 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: 2023-10-30T17:35:02.016 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-10-30T17:35:02.016 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (88017 ms total) 2023-10-30T17:35:02.016 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-10-30T17:35:07.072 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: Running main() from gmock_main.cc 2023-10-30T17:35:07.072 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-10-30T17:35:07.072 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [----------] Global test environment set-up. 2023-10-30T17:35:07.073 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-10-30T17:35:07.073 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-10-30T17:35:07.073 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ OK ] LibRadosStat.Stat (23126 ms) 2023-10-30T17:35:07.073 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-10-30T17:35:07.073 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (2384 ms) 2023-10-30T17:35:07.074 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-10-30T17:35:07.074 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ OK ] LibRadosStat.StatNS (12591 ms) 2023-10-30T17:35:07.074 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-10-30T17:35:07.074 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (287 ms) 2023-10-30T17:35:07.074 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-10-30T17:35:07.075 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (1826 ms) 2023-10-30T17:35:07.075 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [----------] 5 tests from LibRadosStat (40214 ms total) 2023-10-30T17:35:07.075 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: 2023-10-30T17:35:07.075 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-10-30T17:35:07.075 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-10-30T17:35:07.076 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (4064 ms) 2023-10-30T17:35:07.076 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-10-30T17:35:07.076 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (15464 ms) 2023-10-30T17:35:07.076 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-10-30T17:35:07.076 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (4 ms) 2023-10-30T17:35:07.077 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-10-30T17:35:07.077 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1833 ms) 2023-10-30T17:35:07.077 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (21365 ms total) 2023-10-30T17:35:07.077 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: 2023-10-30T17:35:07.077 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [----------] Global test environment tear-down 2023-10-30T17:35:07.078 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (93208 ms total) 2023-10-30T17:35:07.078 INFO:tasks.workunit.client.0.smithi170.stdout: api_stat: [ PASSED ] 9 tests. 2023-10-30T17:35:08.011 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2023-10-30T17:35:08.011 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [----------] Global test environment set-up. 2023-10-30T17:35:08.011 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [----------] 7 tests from LibRadosList 2023-10-30T17:35:08.011 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-10-30T17:35:08.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosList.ListObjects (58343 ms) 2023-10-30T17:35:08.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-10-30T17:35:08.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (17024 ms) 2023-10-30T17:35:08.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-10-30T17:35:08.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: myset foo1,foo2,foo3 2023-10-30T17:35:08.012 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo1 2023-10-30T17:35:08.013 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo2 2023-10-30T17:35:08.013 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo3 2023-10-30T17:35:08.013 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: myset foo1,foo4,foo5 2023-10-30T17:35:08.013 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo4 2023-10-30T17:35:08.013 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo5 2023-10-30T17:35:08.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo1 2023-10-30T17:35:08.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: myset foo6,foo7 2023-10-30T17:35:08.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo7 2023-10-30T17:35:08.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo6 2023-10-30T17:35:08.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-10-30T17:35:08.014 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: ns1:foo4 2023-10-30T17:35:08.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: ns1:foo5 2023-10-30T17:35:08.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: ns2:foo7 2023-10-30T17:35:08.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: ns2:foo6 2023-10-30T17:35:08.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: ns1:foo1 2023-10-30T17:35:08.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: :foo1 2023-10-30T17:35:08.015 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: :foo2 2023-10-30T17:35:08.016 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: :foo3 2023-10-30T17:35:08.016 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (5109 ms) 2023-10-30T17:35:08.016 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-10-30T17:35:08.016 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 1 0 2023-10-30T17:35:08.016 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 10 0 2023-10-30T17:35:08.017 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 13 0 2023-10-30T17:35:08.017 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 7 0 2023-10-30T17:35:08.017 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 14 0 2023-10-30T17:35:08.017 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 0 0 2023-10-30T17:35:08.017 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 15 0 2023-10-30T17:35:08.017 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 11 0 2023-10-30T17:35:08.018 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 5 0 2023-10-30T17:35:08.018 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 8 0 2023-10-30T17:35:08.018 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 6 0 2023-10-30T17:35:08.018 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 3 0 2023-10-30T17:35:08.018 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 4 0 2023-10-30T17:35:08.018 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 12 0 2023-10-30T17:35:08.019 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 9 0 2023-10-30T17:35:08.019 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 2 0 2023-10-30T17:35:08.019 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-10-30T17:35:08.019 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (4240 ms) 2023-10-30T17:35:08.019 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-10-30T17:35:08.020 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: x cursor=MIN 2023-10-30T17:35:08.020 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=1 cursor=7:02547ec2:::1:head 2023-10-30T17:35:08.020 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=10 cursor=7:52ea6a34:::10:head 2023-10-30T17:35:08.020 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=13 cursor=7:566253c9:::13:head 2023-10-30T17:35:08.020 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=7 cursor=7:5c6b0b28:::7:head 2023-10-30T17:35:08.021 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=14 cursor=7:62a1935d:::14:head 2023-10-30T17:35:08.021 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=0 cursor=7:6cac518f:::0:head 2023-10-30T17:35:08.021 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=15 cursor=7:863748b0:::15:head 2023-10-30T17:35:08.021 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=11 cursor=7:89d3ae78:::11:head 2023-10-30T17:35:08.021 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=5 cursor=7:b29083e3:::5:head 2023-10-30T17:35:08.021 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=8 cursor=7:bd63b0f1:::8:head 2023-10-30T17:35:08.022 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=6 cursor=7:c4fdafeb:::6:head 2023-10-30T17:35:08.022 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=3 cursor=7:cfc208b3:::3:head 2023-10-30T17:35:08.022 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=4 cursor=7:d83876eb:::4:head 2023-10-30T17:35:08.022 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=12 cursor=7:de5d7c5f:::12:head 2023-10-30T17:35:08.022 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=9 cursor=7:e960b815:::9:head 2023-10-30T17:35:08.022 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > oid=2 cursor=7:f905c69b:::2:head 2023-10-30T17:35:08.023 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: FIRST> seek to MIN oid=1 2023-10-30T17:35:08.023 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=1 cursor=7:02547ec2:::1:head 2023-10-30T17:35:08.023 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:02547ec2:::1:head 2023-10-30T17:35:08.023 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:02547ec2:::1:head -> 1 2023-10-30T17:35:08.023 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=10 cursor=7:52ea6a34:::10:head 2023-10-30T17:35:08.023 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:52ea6a34:::10:head 2023-10-30T17:35:08.024 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2023-10-30T17:35:08.024 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=13 cursor=7:566253c9:::13:head 2023-10-30T17:35:08.024 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:566253c9:::13:head 2023-10-30T17:35:08.024 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:566253c9:::13:head -> 13 2023-10-30T17:35:08.024 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=7 cursor=7:5c6b0b28:::7:head 2023-10-30T17:35:08.025 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2023-10-30T17:35:08.025 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2023-10-30T17:35:09.968 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=14 cursor=7:62a1935d:::14 api_cmd: Running main() from gmock_main.cc 2023-10-30T17:35:09.968 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-10-30T17:35:09.969 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [----------] Global test environment set-up. 2023-10-30T17:35:09.969 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-10-30T17:35:09.969 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-10-30T17:35:09.969 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (758 ms) 2023-10-30T17:35:09.969 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-10-30T17:35:09.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (79 ms) 2023-10-30T17:35:09.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-10-30T17:35:09.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (87975 ms) 2023-10-30T17:35:09.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-10-30T17:35:09.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:34:55.425142+0000 mon.b [INF] from='client.? 172.21.15.170:0/438131606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi170-91531-16"}]: dispatch 2023-10-30T17:35:09.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:34:56.293445+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi170-91863-7"}]': finished 2023-10-30T17:35:09.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:34:58.219898+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi170-91531-16"}]: dispatch 2023-10-30T17:35:09.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:34:57.802224+0000 mon.c [INF] from='client.? 172.21.15.170:0/865632591' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi170-91863-7"}]: dispatch 2023-10-30T17:35:09.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:34:58.496404+0000 mon.c [INF] from='client.4816 172.21.15.170:0/1710585064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi170-91745-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T17:35:09.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:34:59.886680+0000 mon.a [INF] from='client.? 172.21.15.170:0/74267190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi170-91553-22"}]: dispatch 2023-10-30T17:35:09.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:00.463041+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi170-91863-7"}]: dispatch 2023-10-30T17:35:09.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:00.506569+0000 mon.a [INF] from='client.4816 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi170-91745-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T17:35:09.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:00.766726+0000 mon.a [INF] from='client.? 172.21.15.170:0/74267190' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi170-91553-22"}]: dispatch 2023-10-30T17:35:09.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:01.371191+0000 mon.a [INF] from='client.? 172.21.15.170:0/4041432806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi170-91709-10"}]: dispatch 2023-10-30T17:35:09.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:01.376045+0000 mon.a [INF] from='client.? 172.21.15.170:0/74267190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi170-91553-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T17:35:09.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:01.813361+0000 mon.b [INF] from='client.? 172.21.15.170:0/438131606' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi170-91531-16"}]: dispatch 2023-10-30T17:35:09.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:01.954174+0000 mon.b [INF] from='client.? 172.21.15.170:0/2169515834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi170-91604-10"}]: dispatch 2023-10-30T17:35:10.480 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:03.455671+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=:head 2023-10-30T17:35:10.481 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:62a1935d:::14:head 2023-10-30T17:35:10.481 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:62a1935d:::14:head -> 14 2023-10-30T17:35:10.481 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=0 cursor=7:6cac518f:::0:head 2023-10-30T17:35:10.482 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:6cac518f:::0:head 2023-10-30T17:35:10.482 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:6cac518f:::0:head -> 0 2023-10-30T17:35:10.482 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=15 cursor=7:863748b0:::15:head 2023-10-30T17:35:10.482 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:863748b0:::15:head 2023-10-30T17:35:10.482 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:863748b0:::15:head -> 15 2023-10-30T17:35:10.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=11 cursor=7:89d3ae78:::11:head 2023-10-30T17:35:10.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:89d3ae78:::11:head 2023-10-30T17:35:10.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2023-10-30T17:35:10.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=5 cursor=7:b29083e3:::5:head 2023-10-30T17:35:10.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:b29083e3:::5:head 2023-10-30T17:35:10.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:b29083e3:::5:head -> 5 2023-10-30T17:35:10.484 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=8 cursor=7:bd63b0f1:::8:head 2023-10-30T17:35:10.484 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2023-10-30T17:35:10.484 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2023-10-30T17:35:10.484 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=6 cursor=7:c4fdafeb:::6:head 2023-10-30T17:35:10.484 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2023-10-30T17:35:10.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2023-10-30T17:35:10.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=3 cursor=7:cfc208b3:::3:head 2023-10-30T17:35:10.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:cfc208b3:::3:head 2023-10-30T17:35:10.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2023-10-30T17:35:10.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=4 cursor=7:d83876eb:::4:head 2023-10-30T17:35:10.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:d83876eb:::4:head 2023-10-30T17:35:10.486 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:d83876eb:::4:head -> 4 2023-10-30T17:35:10.486 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=12 cursor=7:de5d7c5f:::12:head 2023-10-30T17:35:10.486 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2023-10-30T17:35:10.486 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2023-10-30T17:35:10.486 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=9 cursor=7:e960b815:::9:head 2023-10-30T17:35:10.487 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:e960b815:::9:head 2023-10-30T17:35:10.487 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:e960b815:::9:head -> 9 2023-10-30T17:35:10.487 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : oid=2 cursor=7:f905c69b:::2:head 2023-10-30T17:35:10.487 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:f905c69b:::2:head 2023-10-30T17:35:10.487 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:f905c69b:::2:head -> 2 2023-10-30T17:35:10.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:566253c9:::13:head 2023-10-30T17:35:10.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:566253c9:::13:head expected=7:566253c9:::13:head 2023-10-30T17:35:10.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:566253c9:::13:head -> 13 2023-10-30T17:35:10.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=13 expected=13 2023-10-30T17:35:10.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:52ea6a34:::10:head 2023-10-30T17:35:10.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:52ea6a34:::10:head expected=7:52ea6a34:::10:head 2023-10-30T17:35:10.489 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2023-10-30T17:35:10.489 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=10 expected=10 2023-10-30T17:35:10.489 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:cfc208b3:::3:head 2023-10-30T17:35:10.489 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:cfc208b3:::3:head expected=7:cfc208b3:::3:head 2023-10-30T17:35:10.489 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2023-10-30T17:35:10.489 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=3 expected=3 2023-10-30T17:35:10.490 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2023-10-30T17:35:10.490 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:c4fdafeb:::6:head expected=7:c4fdafeb:::6:head 2023-10-30T17:35:10.490 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2023-10-30T17:35:10.490 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=6 expected=6 2023-10-30T17:35:10.490 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2023-10-30T17:35:10.491 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:bd63b0f1:::8:head expected=7:bd63b0f1:::8:head 2023-10-30T17:35:10.491 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2023-10-30T17:35:10.491 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=8 expected=8 2023-10-30T17:35:10.491 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:e960b815:::9:head 2023-10-30T17:35:10.491 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:e960b815:::9:head expected=7:e960b815:::9:head 2023-10-30T17:35:10.492 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:e960b815:::9:head -> 9 2023-10-30T17:35:10.492 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=9 expected=9 2023-10-30T17:35:10.492 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2023-10-30T17:35:10.493 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:de5d7c5f:::12:head expected=7:de5d7c5f:::12:head 2023-10-30T17:35:10.493 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2023-10-30T17:35:10.493 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=12 expected=12 2023-10-30T17:35:10.493 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:d83876eb:::4:head 2023-10-30T17:35:10.493 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:d83876eb:::4:head expected=7:d83876eb:::4:head 2023-10-30T17:35:10.494 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:d83876eb:::4:head -> 4 2023-10-30T17:35:10.494 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=4 expected=4 2023-10-30T17:35:10.494 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:b29083e3:::5:head 2023-10-30T17:35:10.744 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:03:54.710 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:10.753 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:03:54.720 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:11.114 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:03:55.080 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:11.684 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:03:55.650 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:11.814 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:55.753 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:11.836 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:55.775 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:11.871 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:03:55.837 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:11.913 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:55.852 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:11.988 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:03:55.953 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:12.872 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: Running main() from gmock_main.cc 2023-10-30T17:35:12.872 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-10-30T17:35:12.873 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [----------] Global test environment set-up. 2023-10-30T17:35:12.873 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-10-30T17:35:12.873 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-10-30T17:35:12.873 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (21962 ms) 2023-10-30T17:35:12.873 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-10-30T17:35:12.874 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLock.LockShared (2612 ms) 2023-10-30T17:35:12.874 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-10-30T17:35:12.874 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (11334 ms) 2023-10-30T17:35:12.874 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-10-30T17:35:12.874 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (4980 ms) 2023-10-30T17:35:12.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-10-30T17:35:12.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (7798 ms) 2023-10-30T17:35:12.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-10-30T17:35:12.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLock.Unlock (3716 ms) 2023-10-30T17:35:12.875 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-10-30T17:35:12.876 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (5056 ms) 2023-10-30T17:35:12.876 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-10-30T17:35:12.876 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (2421 ms) 2023-10-30T17:35:12.876 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [----------] 8 tests from LibRadosLock (59879 ms total) 2023-10-30T17:35:12.877 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: 2023-10-30T17:35:12.877 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-10-30T17:35:12.877 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-10-30T17:35:12.877 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (7318 ms) 2023-10-30T17:35:12.877 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-10-30T17:35:12.878 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (493 ms) 2023-10-30T17:35:12.878 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-10-30T17:35:12.878 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1583 ms) 2023-10-30T17:35:12.878 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-10-30T17:35:12.878 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1223 ms) 2023-10-30T17:35:12.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-10-30T17:35:12.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (254 ms) 2023-10-30T17:35:12.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-10-30T17:35:12.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (479 ms) 2023-10-30T17:35:12.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-10-30T17:35:12.879 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (2117 ms) 2023-10-30T17:35:12.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-10-30T17:35:12.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (759 ms) 2023-10-30T17:35:12.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (14227 ms total) 2023-10-30T17:35:12.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: 2023-10-30T17:35:12.880 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [----------] Global test environment tear-down 2023-10-30T17:35:12.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (99088 ms total) 2023-10-30T17:35:12.881 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock: [ PASSED ] 16 tests. 2023-10-30T17:35:17.170 INFO:tasks.workunit.client.0.smithi170.stdout:[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi170-91531-16"}]: dispatch 2023-10-30T17:35:17.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:03.538129+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi170-91604-10"}]: dispatch 2023-10-30T17:35:17.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:03.677852+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi170-91863-7"}]': finished 2023-10-30T17:35:17.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:03.680188+0000 mon.a [INF] from='client.4816 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi170-91745-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T17:35:17.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:04.701404+0000 mon.a [WRN] Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T17:35:17.171 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:04.685930+0000 mon.c [INF] from='client.? 172.21.15.170:0/1498829829' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-10-30T17:35:17.172 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:05.188748+0000 mon.b [INF] from='client.? 172.21.15.170:0/438131606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi170-91531-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T17:35:17.172 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:05.419311+0000 mon.c [INF] from='client.? 172.21.15.170:0/599189255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi170-91518-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T17:35:17.172 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:06.215378+0000 mon.b [INF] from='client.? 172.21.15.170:0/1066567561' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi170-91495-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T17:35:17.172 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:06.374445+0000 mon.c [INF] from='client.? 172.21.15.170:0/3532982975' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi170-92143-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T17:35:17.173 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:06.477869+0000 client.admin [INF] onexx 2023-10-30T17:35:17.173 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:06.509963+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-10-30T17:35:17.173 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:06.543878+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi170-91518-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T17:35:17.173 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:06.872686+0000 mon.b [INF] from='client.? 172.21.15.170:0/3712709921' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi170-92010-4"}]: dispatch 2023-10-30T17:35:17.173 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:07.010967+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi170-91531-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T17:35:17.174 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:07.019352+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi170-91495-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T17:35:17.174 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:07.050122+0000 mon.a [INF] from='client.? 172.21.15.170:0/4041432806' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi170-91709-10"}]': finished 2023-10-30T17:35:21.096 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:07.052507+0000 mon.a [INF] from='client.? 172.21.15.170:0/74267190' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profil api_lock_pp: Running main() from gmock_main.cc 2023-10-30T17:35:21.097 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-10-30T17:35:21.097 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-10-30T17:35:21.097 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-10-30T17:35:21.097 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: seed 91604 2023-10-30T17:35:21.097 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-10-30T17:35:21.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (17599 ms) 2023-10-30T17:35:21.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-10-30T17:35:21.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (7739 ms) 2023-10-30T17:35:21.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-10-30T17:35:21.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (3234 ms) 2023-10-30T17:35:21.099 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-10-30T17:35:21.099 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (6871 ms) 2023-10-30T17:35:21.099 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-10-30T17:35:21.099 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (9082 ms) 2023-10-30T17:35:21.099 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-10-30T17:35:21.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (5983 ms) 2023-10-30T17:35:21.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-10-30T17:35:21.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (3136 ms) 2023-10-30T17:35:21.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-10-30T17:35:21.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (2945 ms) 2023-10-30T17:35:21.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (56589 ms total) 2023-10-30T17:35:21.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: 2023-10-30T17:35:21.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-10-30T17:35:21.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-10-30T17:35:21.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (6548 ms) 2023-10-30T17:35:21.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-10-30T17:35:21.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (5580 ms) 2023-10-30T17:35:21.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-10-30T17:35:21.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (2460 ms) 2023-10-30T17:35:21.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-10-30T17:35:21.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1319 ms) 2023-10-30T17:35:21.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-10-30T17:35:21.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (193 ms) 2023-10-30T17:35:21.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-10-30T17:35:21.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (234 ms) 2023-10-30T17:35:21.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-10-30T17:35:21.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (313 ms) 2023-10-30T17:35:21.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-10-30T17:35:21.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (189 ms) 2023-10-30T17:35:21.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (16836 ms total) 2023-10-30T17:35:21.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: 2023-10-30T17:35:21.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-10-30T17:35:21.105 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (107351 ms total) 2023-10-30T17:35:21.105 INFO:tasks.workunit.client.0.smithi170.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-10-30T17:35:22.369 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_1_[92301]: starting. 2023-10-30T17:35:22.369 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_1_[92301]: creating pool ceph_test_rados_list_parallel.smithi170-92229 2023-10-30T17:35:22.370 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_1_[92301]: created object 0... 2023-10-30T17:35:22.370 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_1_[92301]: created object 25... 2023-10-30T17:35:22.370 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_1_[92301]: created object 49... 2023-10-30T17:35:22.370 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_1_[92301]: finishing. 2023-10-30T17:35:22.370 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_1_[92301]: shutting down. 2023-10-30T17:35:22.371 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_2_[92302]: starting. 2023-10-30T17:35:22.371 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_2_[92302]: listing objects. 2023-10-30T17:35:22.371 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_2_[92302]: listed object 0... 2023-10-30T17:35:22.371 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_2_[92302]: listed object 25... 2023-10-30T17:35:22.371 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_2_[92302]: saw 50 objects 2023-10-30T17:35:22.372 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_2_[92302]: shutting down. 2023-10-30T17:35:22.372 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:22.372 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_3_[103118]: starting. 2023-10-30T17:35:22.372 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_3_[103118]: creating pool ceph_test_rados_list_parallel.smithi170-92229 2023-10-30T17:35:22.372 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_3_[103118]: created object 0... 2023-10-30T17:35:22.373 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_3_[103118]: created object 25... 2023-10-30T17:35:22.373 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_3_[103118]: created object 49... 2023-10-30T17:35:22.373 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_3_[103118]: finishing. 2023-10-30T17:35:22.373 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_3_[103118]: shutting down. 2023-10-30T17:35:22.373 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:22.373 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_4_[103119]: starting. 2023-10-30T17:35:22.374 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_4_[103119]: listing objects. 2023-10-30T17:35:22.374 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_4_[103119]: listed object 0... 2023-10-30T17:35:22.374 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_4_[103119]: listed object 25... 2023-10-30T17:35:22.374 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_4_[103119]: saw 48 objects 2023-10-30T17:35:22.374 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_4_[103119]: shutting down. 2023-10-30T17:35:22.375 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:22.375 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_5_[103120]: starting. 2023-10-30T17:35:22.375 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_5_[103120]: removed 25 objects... 2023-10-30T17:35:22.375 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_5_[103120]: removed half of the objects 2023-10-30T17:35:22.375 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_5_[103120]: removed 50 objects... 2023-10-30T17:35:22.376 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_5_[103120]: removed 50 objects 2023-10-30T17:35:22.376 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_5_[103120]: shutting down. 2023-10-30T17:35:22.376 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:22.376 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:22.376 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_6_[113228]: starting. 2023-10-30T17:35:22.377 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_6_[113228]: creating pool ceph_test_rados_list_parallel.smithi170-92229 2023-10-30T17:35:22.377 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_6_[113228]: created object 0... 2023-10-30T17:35:22.377 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_6_[113228]: created object 25... 2023-10-30T17:35:22.377 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_6_[113228]: created object 49... 2023-10-30T17:35:22.377 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_6_[113228]: finishing. 2023-10-30T17:35:22.378 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_6_[113228]: shutting down. 2023-10-30T17:35:22.378 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:22.378 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:22.378 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_7_[113229]: starting. 2023-10-30T17:35:22.378 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_7_[113229]: listing objects. 2023-10-30T17:35:22.379 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_7_[113229]: listed object 0... 2023-10-30T17:35:22.379 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_7_[113229]: listed object 25... 2023-10-30T17:35:22.379 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_7_[113229]: listed object 50... 2023-10-30T17:35:22.379 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_7_[113229]: saw 51 objects 2023-10-30T17:35:22.379 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_7_[113229]: shutting down. 2023-10-30T17:35:22.379 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:22.380 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:22.380 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_8_[113231]: starting. 2023-10-30T17:35:22.380 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_8_[113231]: added 25 objects... 2023-10-30T17:35:22.380 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_8_[113231]: added half of the objects 2023-10-30T17:35:22.380 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_8_[113231]: added 50 objects... 2023-10-30T17:35:22.381 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_8_[113231]: added 50 objects 2023-10-30T17:35:22.381 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_8_[113231]: shutting down. 2023-10-30T17:35:22.381 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:22.381 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:26.808 INFO:tasks.workunit.client.0.smithi170.stdout: e set", "name": "testprofile-LibRadosIoECPP_smithi170-91553-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T17:35:26.808 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:07.054940+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi170-91604-10"}]': finished 2023-10-30T17:35:26.809 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:07.136734+0000 mon.c [INF] from='client.4816 172.21.15.170:0/1710585064' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi170-91745-4", "tierpool": "test-rados-api-smithi170-91745-4-cache"}]: dispatch 2023-10-30T17:35:26.809 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:07.565876+0000 mon.a [INF] from='client.? 172.21.15.170:0/4041432806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi170-91709-10"}]: dispatch 2023-10-30T17:35:26.809 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:08.096362+0000 mon.a [INF] from='client.? 172.21.15.170:0/74267190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi170-91553-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi170-91553-22"}]: dispatch 2023-10-30T17:35:26.810 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:08.746130+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi170-92010-4"}]: dispatch 2023-10-30T17:35:26.810 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:09.107116+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi170-92143-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T17:35:26.810 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:09.211617+0000 mon.a [INF] from='client.4816 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi170-91745-4", "tierpool": "test-rados-api-smithi170-91745-4-cache"}]: dispatch 2023-10-30T17:35:26.810 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:13.017570+0000 mon.b [INF] from='client.? 172.21.15.170:0/438131606' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi170-91531-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi170-91531-16"}]: dispatch 2023-10-30T17:35:26.811 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:16.544891+0000 mon.b [INF] from='client.? 172.21.15.170:0/3058399550' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi170-91975-12"}]: dispatch 2023-10-30T17:35:26.811 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:16.618648+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi170-91531-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi170-91531-16"}]: dispatch 2023-10-30T17:35:26.811 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:17.414628+0000 mon.c [INF] from='client.? 172.21.15.170:0/1498829829' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-10-30T17:35:26.811 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:17.596860+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi170-92010-4"}]': finished 2023-10-30T17:35:26.812 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:17.598691+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi170-92143-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T17:35:26.812 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:17.600622+0000 mon.a [INF] from='client.4816 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi170-91745-4", "tierpool": "test-rados-api-smithi170-91745-4-cache"}]': finished 2023-10-30T17:35:32.564 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:17.602526+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfMana api_io_pp: Running main() from gmock_main.cc 2023-10-30T17:35:32.564 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-10-30T17:35:32.564 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [----------] Global test environment set-up. 2023-10-30T17:35:32.564 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-10-30T17:35:32.565 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: seed 91553 2023-10-30T17:35:32.565 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-10-30T17:35:32.565 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-10-30T17:35:32.565 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-10-30T17:35:32.565 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (23010 ms) 2023-10-30T17:35:32.566 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-10-30T17:35:32.566 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (10923 ms) 2023-10-30T17:35:32.566 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-10-30T17:35:32.566 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (2135 ms) 2023-10-30T17:35:32.566 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-10-30T17:35:32.567 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (5331 ms) 2023-10-30T17:35:32.567 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-10-30T17:35:32.567 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (6011 ms) 2023-10-30T17:35:32.567 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-10-30T17:35:32.567 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (189 ms) 2023-10-30T17:35:32.568 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-10-30T17:35:32.568 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (963 ms) 2023-10-30T17:35:32.568 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-10-30T17:35:32.568 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (3959 ms) 2023-10-30T17:35:32.568 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-10-30T17:35:32.569 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (1288 ms) 2023-10-30T17:35:32.569 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-10-30T17:35:32.569 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (68 ms) 2023-10-30T17:35:32.569 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-10-30T17:35:32.569 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (3052 ms) 2023-10-30T17:35:32.570 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-10-30T17:35:32.570 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (2454 ms) 2023-10-30T17:35:32.570 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-10-30T17:35:32.570 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (3022 ms) 2023-10-30T17:35:32.570 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-10-30T17:35:32.571 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3019 ms) 2023-10-30T17:35:32.571 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-10-30T17:35:32.571 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (6847 ms) 2023-10-30T17:35:32.571 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-10-30T17:35:32.571 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (261 ms) 2023-10-30T17:35:32.571 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-10-30T17:35:32.572 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (447 ms) 2023-10-30T17:35:32.572 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-10-30T17:35:32.572 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (433 ms) 2023-10-30T17:35:32.572 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-10-30T17:35:32.572 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (106 ms) 2023-10-30T17:35:32.573 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (73518 ms total) 2023-10-30T17:35:32.573 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: 2023-10-30T17:35:32.573 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-10-30T17:35:32.573 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-10-30T17:35:32.573 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (9181 ms) 2023-10-30T17:35:32.574 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-10-30T17:35:32.574 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (523 ms) 2023-10-30T17:35:32.574 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-10-30T17:35:32.574 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (342 ms) 2023-10-30T17:35:32.574 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-10-30T17:35:32.575 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (234 ms) 2023-10-30T17:35:32.575 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-10-30T17:35:36.043 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundgedPP_smithi170-91883-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T17:35:36.043 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:17.604561+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi170-91604-10"}]': finished 2023-10-30T17:35:36.043 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:19.465897+0000 client.admin [INF] twoxx 2023-10-30T17:35:36.044 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:19.494859+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-10-30T17:35:36.044 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:20.129194+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi170-91975-12"}]: dispatch 2023-10-30T17:35:36.044 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:19.257123+0000 mon.c [INF] from='client.4816 172.21.15.170:0/1710585064' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi170-91745-4", "overlaypool": "test-rados-api-smithi170-91745-4-cache"}]: dispatch 2023-10-30T17:35:36.044 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:19.697993+0000 mon.b [INF] from='client.? 172.21.15.170:0/3712709921' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi170-92010-4", "tierpool": "test-rados-api-smithi170-92010-6"}]: dispatch 2023-10-30T17:35:36.044 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:21.028534+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi170-92010-4", "tierpool": "test-rados-api-smithi170-92010-6"}]: dispatch 2023-10-30T17:35:36.045 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:21.265852+0000 mon.a [INF] from='client.4816 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi170-91745-4", "overlaypool": "test-rados-api-smithi170-91745-4-cache"}]: dispatch 2023-10-30T17:35:36.045 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:21.355446+0000 mon.a [INF] from='client.? 172.21.15.170:0/74267190' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi170-91553-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi170-91553-22"}]': finished 2023-10-30T17:35:36.045 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:22.377006+0000 mon.a [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-10-30T17:35:36.045 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:22.858003+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-10-30T17:35:36.046 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:23.200183+0000 mon.b [INF] from='client.? 172.21.15.170:0/3058399550' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi170-91975-12"}]: dispatch 2023-10-30T17:35:36.046 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:24.239360+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi170-91975-12"}]: dispatch 2023-10-30T17:35:36.046 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:24.448626+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi170-91531-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi170-91531-16"}]': finished 2023-10-30T17:35:36.046 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:24.456365+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi170-92010-4", "tierpool": "test-rados-api-smithi170-92010-6"}]': finished 2023-10-30T17:35:36.047 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:24.458588+0000 mon.a [INF] from='client.4816 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi170-91745-4", "overlaypool": "test-rados-api-smithi170-91745-4-cache"}]': finished 2023-10-30T17:35:36.542 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:26.122201+0000 mon.b [INF] from='client.? 172.21.15.170:0/3058399550' list_parallel: ******************************* 2023-10-30T17:35:36.542 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_9_[121193]: starting. 2023-10-30T17:35:36.542 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_9_[121193]: creating pool ceph_test_rados_list_parallel.smithi170-92229 2023-10-30T17:35:36.542 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_9_[121193]: created object 0... 2023-10-30T17:35:36.543 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_9_[121193]: created object 25... 2023-10-30T17:35:36.543 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_9_[121193]: created object 49... 2023-10-30T17:35:36.543 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_9_[121193]: finishing. 2023-10-30T17:35:36.543 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_9_[121193]: shutting down. 2023-10-30T17:35:36.543 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.544 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.544 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.544 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_10_[121194]: starting. 2023-10-30T17:35:36.544 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_10_[121194]: listing objects. 2023-10-30T17:35:36.544 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_10_[121194]: listed object 0... 2023-10-30T17:35:36.545 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_10_[121194]: listed object 25... 2023-10-30T17:35:36.545 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_10_[121194]: listed object 50... 2023-10-30T17:35:36.545 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_10_[121194]: listed object 75... 2023-10-30T17:35:36.545 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_10_[121194]: saw 97 objects 2023-10-30T17:35:36.546 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_10_[121194]: shutting down. 2023-10-30T17:35:36.546 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.546 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.546 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.546 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_13_[121197]: starting. 2023-10-30T17:35:36.547 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_13_[121197]: removed 25 objects... 2023-10-30T17:35:36.547 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_13_[121197]: removed half of the objects 2023-10-30T17:35:36.547 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_13_[121197]: removed 50 objects... 2023-10-30T17:35:36.548 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_13_[121197]: removed 50 objects 2023-10-30T17:35:36.548 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_13_[121197]: shutting down. 2023-10-30T17:35:36.548 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.548 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.548 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.549 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_12_[121196]: starting. 2023-10-30T17:35:36.549 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_12_[121196]: added 25 objects... 2023-10-30T17:35:36.549 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_12_[121196]: added half of the objects 2023-10-30T17:35:36.549 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_12_[121196]: added 50 objects... 2023-10-30T17:35:36.549 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_12_[121196]: added 50 objects 2023-10-30T17:35:36.549 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_12_[121196]: shutting down. 2023-10-30T17:35:36.550 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.550 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.550 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.550 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_11_[121195]: starting. 2023-10-30T17:35:36.551 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_11_[121195]: added 25 objects... 2023-10-30T17:35:36.551 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_11_[121195]: added half of the objects 2023-10-30T17:35:36.551 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_11_[121195]: added 50 objects... 2023-10-30T17:35:36.551 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_11_[121195]: added 50 objects 2023-10-30T17:35:36.551 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_11_[121195]: shutting down. 2023-10-30T17:35:36.552 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.552 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.552 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.552 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.552 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_14_[131353]: starting. 2023-10-30T17:35:36.552 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_14_[131353]: creating pool ceph_test_rados_list_parallel.smithi170-92229 2023-10-30T17:35:36.553 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_14_[131353]: created object 0... 2023-10-30T17:35:36.553 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_14_[131353]: created object 25... 2023-10-30T17:35:36.553 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_14_[131353]: created object 49... 2023-10-30T17:35:36.553 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_14_[131353]: finishing. 2023-10-30T17:35:36.553 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_14_[131353]: shutting down. 2023-10-30T17:35:36.554 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.554 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.554 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.554 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:36.555 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_15_[131354]: starting. 2023-10-30T17:35:36.555 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_15_[131354]: listing objects. 2023-10-30T17:35:40.991 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_15_[131354]: open_pools_parallel: process_1_[92285]: starting. 2023-10-30T17:35:40.991 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_1_[92285]: creating pool ceph_test_rados_open_pools_parallel.smithi170-92226 2023-10-30T17:35:40.991 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_1_[92285]: created object 0... 2023-10-30T17:35:40.991 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_1_[92285]: created object 25... 2023-10-30T17:35:40.992 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_1_[92285]: created object 49... 2023-10-30T17:35:40.992 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_1_[92285]: finishing. 2023-10-30T17:35:40.992 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_1_[92285]: shutting down. 2023-10-30T17:35:40.992 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_2_[92298]: starting. 2023-10-30T17:35:40.992 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_2_[92298]: rados_pool_create. 2023-10-30T17:35:40.993 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_2_[92298]: rados_ioctx_create. 2023-10-30T17:35:40.993 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_2_[92298]: shutting down. 2023-10-30T17:35:40.993 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: ******************************* 2023-10-30T17:35:40.993 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_3_[111183]: starting. 2023-10-30T17:35:40.993 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_3_[111183]: creating pool ceph_test_rados_open_pools_parallel.smithi170-92226 2023-10-30T17:35:40.994 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_3_[111183]: created object 0... 2023-10-30T17:35:40.994 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_3_[111183]: created object 25... 2023-10-30T17:35:40.994 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_3_[111183]: created object 49... 2023-10-30T17:35:40.994 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_3_[111183]: finishing. 2023-10-30T17:35:40.994 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_3_[111183]: shutting down. 2023-10-30T17:35:40.994 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: ******************************* 2023-10-30T17:35:40.995 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_4_[111184]: starting. 2023-10-30T17:35:40.995 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_4_[111184]: rados_pool_create. 2023-10-30T17:35:40.995 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_4_[111184]: rados_ioctx_create. 2023-10-30T17:35:40.995 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: process_4_[111184]: shutting down. 2023-10-30T17:35:40.995 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: ******************************* 2023-10-30T17:35:40.996 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: ******************************* 2023-10-30T17:35:40.996 INFO:tasks.workunit.client.0.smithi170.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-10-30T17:35:45.663 INFO:tasks.workunit.client.0.smithi170.stdout:hing 2023-10-30T17:35:45.664 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: flushed 2023-10-30T17:35:45.664 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3293 ms) 2023-10-30T17:35:45.664 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-10-30T17:35:45.665 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: trying... 2023-10-30T17:35:45.665 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: handle_notify cookie 94034627366784 notify_id 382252089346 notifier_gid 4735 2023-10-30T17:35:45.665 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: timed out 2023-10-30T17:35:45.665 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: flushing 2023-10-30T17:35:45.665 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: flushed 2023-10-30T17:35:45.666 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3159 ms) 2023-10-30T17:35:45.666 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-10-30T17:35:45.666 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: List watches 2023-10-30T17:35:45.666 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: notify2 2023-10-30T17:35:45.666 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: handle_notify cookie 94034627367824 notify_id 386547056642 notifier_gid 4735 2023-10-30T17:35:45.667 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: notify2 done 2023-10-30T17:35:45.667 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: watch_check 2023-10-30T17:35:45.667 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: unwatch2 2023-10-30T17:35:45.667 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: flushing 2023-10-30T17:35:45.667 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: done 2023-10-30T17:35:45.668 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3285 ms) 2023-10-30T17:35:45.668 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-10-30T17:35:45.668 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: List watches 2023-10-30T17:35:45.668 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: notify2 2023-10-30T17:35:45.668 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: handle_notify cookie 94034627367824 notify_id 390842023939 notifier_gid 4735 2023-10-30T17:35:45.669 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: notify2 done 2023-10-30T17:35:45.669 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: watch_check 2023-10-30T17:35:45.669 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: unwatch2 2023-10-30T17:35:45.669 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: flushing 2023-10-30T17:35:45.669 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: done 2023-10-30T17:35:45.669 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3384 ms) 2023-10-30T17:35:45.670 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (51301 ms total) 2023-10-30T17:35:45.670 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: 2023-10-30T17:35:45.670 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-10-30T17:35:45.670 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (131738 ms total) 2023-10-30T17:35:45.670 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-10-30T17:35:45.671 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_1_[92325]: starting. 2023-10-30T17:35:45.671 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_1_[92325]: creating pool ceph_test_rados_delete_pools_parallel.smithi170-92303 2023-10-30T17:35:45.671 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_1_[92325]: created object 0... 2023-10-30T17:35:45.672 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_1_[92325]: created object 25... 2023-10-30T17:35:45.672 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_1_[92325]: created object 49... 2023-10-30T17:35:45.672 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_1_[92325]: finishing. 2023-10-30T17:35:45.672 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_1_[92325]: shutting down. 2023-10-30T17:35:45.672 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_2_[92327]: starting. 2023-10-30T17:35:45.673 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_2_[92327]: deleting pool ceph_test_rados_delete_pools_parallel.smithi170-92303 2023-10-30T17:35:45.673 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_2_[92327]: shutting down. 2023-10-30T17:35:45.673 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: ******************************* 2023-10-30T17:35:45.673 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_3_[108085]: starting. 2023-10-30T17:35:45.673 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_3_[108085]: creating pool ceph_test_rados_delete_pools_parallel.smithi170-92303 2023-10-30T17:35:45.674 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_3_[108085]: created object 0... 2023-10-30T17:35:45.674 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_3_[108085]: created object 25... 2023-10-30T17:35:45.674 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_3_[108085]: created object 49... 2023-10-30T17:35:45.674 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_3_[108085]: finishing. 2023-10-30T17:35:45.674 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_3_[108085]: shutting down. 2023-10-30T17:35:45.675 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: ******************************* 2023-10-30T17:35:45.675 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_5_[108087]: starting. 2023-10-30T17:35:45.675 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_5_[108087]: listing objects. 2023-10-30T17:35:45.675 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_5_[108087]: listed object 0... 2023-10-30T17:35:45.676 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_5_[108087]: listed object 25... 2023-10-30T17:35:45.676 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_5_[108087]: saw 50 objects 2023-10-30T17:35:45.676 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_5_[108087]: shutting down. 2023-10-30T17:35:45.676 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: ******************************* 2023-10-30T17:35:45.677 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_4_[108086]: starting. 2023-10-30T17:35:45.677 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_4_[108086]: deleting pool ceph_test_rados_delete_pools_parallel.smithi170-92303 2023-10-30T17:35:45.677 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: process_4_[108086]: shutting down. 2023-10-30T17:35:45.677 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: ******************************* 2023-10-30T17:35:45.677 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: ******************************* 2023-10-30T17:35:45.678 INFO:tasks.workunit.client.0.smithi170.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-10-30T17:35:48.580 INFO:tasks.workunit.client.0.smithi170.stdout: entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi170-91975-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T17:35:48.580 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:26.296149+0000 mon.b [INF] from='client.? 172.21.15.170:0/3712709921' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi170-92010-6", "pool2": "test-rados-api-smithi170-92010-6", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-30T17:35:48.580 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:26.455335+0000 mon.c [INF] from='client.4816 172.21.15.170:0/1710585064' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi170-91745-4-cache", "mode": "writeback"}]: dispatch 2023-10-30T17:35:48.581 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:26.579701+0000 mon.c [INF] from='client.? 172.21.15.170:0/1498829829' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-10-30T17:35:48.581 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:27.895147+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi170-91975-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T17:35:48.581 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:27.899944+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi170-92010-6", "pool2": "test-rados-api-smithi170-92010-6", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-30T17:35:48.581 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:28.759945+0000 client.admin [INF] threexx 2023-10-30T17:35:48.581 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:28.788930+0000 mon.a [INF] from='client.4816 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi170-91745-4-cache", "mode": "writeback"}]: dispatch 2023-10-30T17:35:48.582 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:28.793461+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-10-30T17:35:48.582 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:29.581585+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-10-30T17:35:48.582 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: got: 2023-10-30T17:35:29.584397+0000 mon.a [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T17:35:48.582 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (45846 ms) 2023-10-30T17:35:48.582 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (134658 ms total) 2023-10-30T17:35:48.583 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: 2023-10-30T17:35:48.583 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [----------] Global test environment tear-down 2023-10-30T17:35:48.583 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (134658 ms total) 2023-10-30T17:35:48.583 INFO:tasks.workunit.client.0.smithi170.stdout: api_cmd: [ PASSED ] 4 tests. 2023-10-30T17:35:48.894 INFO:tasks.workunit.client.0.smithi170.stdout: listed object 0... 2023-10-30T17:35:48.894 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_15_[131354]: listed object 25... 2023-10-30T17:35:48.894 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_15_[131354]: listed object 50... 2023-10-30T17:35:48.895 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_15_[131354]: listed object 75... 2023-10-30T17:35:48.895 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_15_[131354]: listed object 100... 2023-10-30T17:35:48.895 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_15_[131354]: listed object 125... 2023-10-30T17:35:48.895 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_15_[131354]: saw 150 objects 2023-10-30T17:35:48.896 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_15_[131354]: shutting down. 2023-10-30T17:35:48.896 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:48.896 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:48.896 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:48.896 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:48.897 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_16_[131356]: starting. 2023-10-30T17:35:48.897 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_16_[131356]: added 25 objects... 2023-10-30T17:35:48.897 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_16_[131356]: added half of the objects 2023-10-30T17:35:48.897 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_16_[131356]: added 50 objects... 2023-10-30T17:35:48.897 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_16_[131356]: added 50 objects 2023-10-30T17:35:48.897 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: process_16_[131356]: shutting down. 2023-10-30T17:35:48.898 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:48.898 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:48.898 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:48.898 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:48.898 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******************************* 2023-10-30T17:35:48.899 INFO:tasks.workunit.client.0.smithi170.stdout: list_parallel: ******* SUCCESS ********** 2023-10-30T17:35:52.354 INFO:tasks.workunit.client.0.smithi170.stdout:TripPP2 (313 ms) 2023-10-30T17:35:52.354 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-10-30T17:35:52.354 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (134 ms) 2023-10-30T17:35:52.355 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-10-30T17:35:52.355 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (462 ms) 2023-10-30T17:35:52.355 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-10-30T17:35:52.355 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (101 ms) 2023-10-30T17:35:52.355 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-10-30T17:35:52.355 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (258 ms) 2023-10-30T17:35:52.356 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-10-30T17:35:52.356 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (134 ms) 2023-10-30T17:35:52.356 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-10-30T17:35:52.356 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (121 ms) 2023-10-30T17:35:52.356 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-10-30T17:35:52.357 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (1535 ms) 2023-10-30T17:35:52.357 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-10-30T17:35:52.357 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (888 ms) 2023-10-30T17:35:52.357 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-10-30T17:35:52.358 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (244 ms) 2023-10-30T17:35:52.358 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-10-30T17:35:52.358 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (241 ms) 2023-10-30T17:35:52.358 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-10-30T17:35:52.358 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (174 ms) 2023-10-30T17:35:52.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-10-30T17:35:52.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (275 ms) 2023-10-30T17:35:52.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (15160 ms total) 2023-10-30T17:35:52.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: 2023-10-30T17:35:52.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [----------] Global test environment tear-down 2023-10-30T17:35:52.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (138630 ms total) 2023-10-30T17:35:52.360 INFO:tasks.workunit.client.0.smithi170.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-10-30T17:35:54.810 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:04:38.776 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:54.842 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:04:38.808 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:54.843 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:38.782 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:54.847 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:04:38.813 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:54.900 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:38.839 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:54.909 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:04:38.875 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:54.964 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:38.903 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:55.026 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:04:38.992 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:55.125 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:04:39.091 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:35:56.096 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: Running main() from gmock_main.cc 2023-10-30T17:35:56.097 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-10-30T17:35:56.097 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [----------] Global test environment set-up. 2023-10-30T17:35:56.097 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-10-30T17:35:56.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-10-30T17:35:56.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (12620 ms) 2023-10-30T17:35:56.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-10-30T17:35:56.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-10-30T17:35:56.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-10-30T17:35:56.098 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (23985 ms) 2023-10-30T17:35:56.099 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-10-30T17:35:56.099 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (2951 ms) 2023-10-30T17:35:56.099 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-10-30T17:35:56.099 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.Checksum (4338 ms) 2023-10-30T17:35:56.099 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-10-30T17:35:56.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (225 ms) 2023-10-30T17:35:56.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-10-30T17:35:56.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (1667 ms) 2023-10-30T17:35:56.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-10-30T17:35:56.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (1277 ms) 2023-10-30T17:35:56.100 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-10-30T17:35:56.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (1596 ms) 2023-10-30T17:35:56.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-10-30T17:35:56.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.TruncTest (2694 ms) 2023-10-30T17:35:56.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-10-30T17:35:56.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (8368 ms) 2023-10-30T17:35:56.101 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-10-30T17:35:56.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (2860 ms) 2023-10-30T17:35:56.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-10-30T17:35:56.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.RmXattr (6462 ms) 2023-10-30T17:35:56.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-10-30T17:35:56.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIo.XattrIter (2764 ms) 2023-10-30T17:35:56.102 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [----------] 14 tests from LibRadosIo (71807 ms total) 2023-10-30T17:35:56.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: 2023-10-30T17:35:56.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-10-30T17:35:56.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-10-30T17:35:56.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (7227 ms) 2023-10-30T17:35:56.103 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-10-30T17:35:56.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (281 ms) 2023-10-30T17:35:56.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-10-30T17:35:56.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (174 ms) 2023-10-30T17:35:56.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-10-30T17:35:56.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (437 ms) 2023-10-30T17:35:56.104 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-10-30T17:35:56.105 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (445 ms) 2023-10-30T17:35:56.105 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-10-30T17:35:56.105 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (219 ms) 2023-10-30T17:35:56.105 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-10-30T17:35:56.105 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (207 ms) 2023-10-30T17:35:56.105 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-10-30T17:35:56.106 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (199 ms) 2023-10-30T17:35:56.106 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-10-30T17:35:56.106 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (455 ms) 2023-10-30T17:35:56.106 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-10-30T17:35:56.106 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (182 ms) 2023-10-30T17:35:56.106 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [----------] 10 tests from LibRadosIoEC (9826 ms total) 2023-10-30T17:35:56.107 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: 2023-10-30T17:35:56.107 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [----------] Global test environment tear-down 2023-10-30T17:35:56.107 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [==========] 24 tests from 2 test suites ran. (142388 ms total) 2023-10-30T17:35:56.107 INFO:tasks.workunit.client.0.smithi170.stdout: api_io: [ PASSED ] 24 tests. 2023-10-30T17:35:56.114 INFO:tasks.workunit.client.0.smithi170.stdout:misc: [----------] 1 test from LibRadosMiscPool (29232 ms total) 2023-10-30T17:35:56.114 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: 2023-10-30T17:35:56.115 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-10-30T17:35:56.115 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-10-30T17:35:56.115 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-10-30T17:35:56.115 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-10-30T17:35:56.115 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMisc.Exec (5713 ms) 2023-10-30T17:35:56.115 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-10-30T17:35:56.116 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (25962 ms) 2023-10-30T17:35:56.116 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-10-30T17:35:56.116 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2399 ms) 2023-10-30T17:35:56.116 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-10-30T17:35:56.116 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4415 ms) 2023-10-30T17:35:56.117 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-10-30T17:35:56.117 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-10-30T17:35:56.117 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-10-30T17:35:56.117 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-10-30T17:35:56.117 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-10-30T17:35:56.117 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (69217 ms) 2023-10-30T17:35:56.118 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [----------] 8 tests from LibRadosMisc (107706 ms total) 2023-10-30T17:35:56.118 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: 2023-10-30T17:35:56.118 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [----------] Global test environment tear-down 2023-10-30T17:35:56.118 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (142312 ms total) 2023-10-30T17:35:56.118 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc: [ PASSED ] 12 tests. 2023-10-30T17:35:58.501 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-10-30T17:35:58.501 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-10-30T17:35:58.501 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-10-30T17:35:58.502 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-10-30T17:35:58.502 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-10-30T17:35:58.502 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T17:35:58.502 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (26336 ms) 2023-10-30T17:35:58.502 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-10-30T17:35:58.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94391179422208 err -107 2023-10-30T17:35:58.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-10-30T17:35:58.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (10306 ms) 2023-10-30T17:35:58.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-10-30T17:35:58.503 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-10-30T17:35:58.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94391179422208 err -107 2023-10-30T17:35:58.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (6764 ms) 2023-10-30T17:35:58.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-10-30T17:35:58.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_cb from 4486 notify_id 279172874240 cookie 94391181785648 2023-10-30T17:35:58.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (6971 ms) 2023-10-30T17:35:58.504 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-10-30T17:35:58.505 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_cb from 4486 notify_id 313532612608 cookie 94391181785648 2023-10-30T17:35:58.505 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (13025 ms) 2023-10-30T17:35:58.505 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-10-30T17:35:58.505 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_cb from 4486 notify_id 365072220161 cookie 94391181861696 2023-10-30T17:35:58.505 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (5843 ms) 2023-10-30T17:35:58.506 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-10-30T17:35:58.506 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_cb from 4486 notify_id 373662154755 cookie 94391181861696 2023-10-30T17:35:58.506 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_cb from 4486 notify_id 373662154755 cookie 94391181912160 2023-10-30T17:35:58.506 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (2589 ms) 2023-10-30T17:35:58.506 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-10-30T17:35:58.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_cb from 4486 notify_id 377957122049 cookie 94391181912160 2023-10-30T17:35:58.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_cb from 4486 notify_id 377957122050 cookie 94391181912160 2023-10-30T17:35:58.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_cb from 4486 notify_id 377957122050 cookie 94391181912160 2023-10-30T17:35:58.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_cb from 4486 notify_id 377957122051 cookie 94391181912160 2023-10-30T17:35:58.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (5663 ms) 2023-10-30T17:35:58.507 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-10-30T17:35:58.508 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-10-30T17:35:58.508 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94391181912160 err -107 2023-10-30T17:35:58.508 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_cb from 4486 notify_id 386547056641 cookie 94391181912160 2023-10-30T17:35:58.508 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5438 ms) 2023-10-30T17:35:58.508 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-10-30T17:35:58.508 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-10-30T17:35:58.509 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94391181912160 err -107 2023-10-30T17:35:58.509 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1946 ms) 2023-10-30T17:35:58.509 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (84881 ms total) 2023-10-30T17:35:58.509 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: 2023-10-30T17:35:58.509 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-10-30T17:35:58.510 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-10-30T17:35:58.510 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T17:35:58.510 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (4738 ms) 2023-10-30T17:35:58.510 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (4738 ms total) 2023-10-30T17:35:58.510 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: 2023-10-30T17:35:58.510 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-10-30T17:35:58.511 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (144605 ms total) 2023-10-30T17:35:58.511 INFO:tasks.workunit.client.0.smithi170.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-10-30T17:36:27.288 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: Running main() from gmock_main.cc 2023-10-30T17:36:27.288 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-10-30T17:36:27.288 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [----------] Global test environment set-up. 2023-10-30T17:36:27.289 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-10-30T17:36:27.289 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-10-30T17:36:27.289 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (19921 ms) 2023-10-30T17:36:27.289 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-10-30T17:36:27.289 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (8406 ms) 2023-10-30T17:36:27.290 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-10-30T17:36:27.290 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (14337 ms) 2023-10-30T17:36:27.290 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-10-30T17:36:27.290 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (5091 ms) 2023-10-30T17:36:27.290 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (47755 ms total) 2023-10-30T17:36:27.291 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: 2023-10-30T17:36:27.291 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-10-30T17:36:27.291 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-10-30T17:36:27.291 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (10908 ms) 2023-10-30T17:36:27.291 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-10-30T17:36:27.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (9022 ms) 2023-10-30T17:36:27.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-10-30T17:36:27.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (53528 ms) 2023-10-30T17:36:27.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (73458 ms total) 2023-10-30T17:36:27.292 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: 2023-10-30T17:36:27.293 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-10-30T17:36:27.293 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-10-30T17:36:27.293 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2898 ms) 2023-10-30T17:36:27.293 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-10-30T17:36:27.293 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2174 ms) 2023-10-30T17:36:27.294 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-10-30T17:36:27.294 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2799 ms) 2023-10-30T17:36:27.294 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-10-30T17:36:27.294 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2162 ms) 2023-10-30T17:36:27.294 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (10033 ms total) 2023-10-30T17:36:27.295 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: 2023-10-30T17:36:27.295 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-10-30T17:36:27.295 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-10-30T17:36:27.295 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (6297 ms) 2023-10-30T17:36:27.295 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-10-30T17:36:27.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4223 ms) 2023-10-30T17:36:27.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (10521 ms total) 2023-10-30T17:36:27.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: 2023-10-30T17:36:27.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [----------] Global test environment tear-down 2023-10-30T17:36:27.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (173449 ms total) 2023-10-30T17:36:27.296 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-10-30T17:36:39.293 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:05:23.259 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:39.309 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:05:23.274 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:39.316 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:05:23.282 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:39.317 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:05:23.283 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:39.467 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:05:23.433 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:39.470 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:23.409 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:39.481 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:23.420 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:39.500 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:05:23.466 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:39.721 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:23.660 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:59.483 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:05:43.448 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:59.535 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:05:43.501 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:59.578 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:43.517 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:59.583 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:05:43.549 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:59.586 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:43.525 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:59.588 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:05:43.554 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:59.595 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:05:43.562 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:59.629 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:43.568 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:36:59.636 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:05:43.602 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:11.296 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:37:11.290+0000 10a5c640 -1 osd.7 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4425.0:2023 12.e 12:73912ba8:test-rados-api-smithi170-91706-15::foo:head [setxattr a987 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148) 2023-10-30T17:37:12.328 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:37:12.326+0000 10a5c640 -1 osd.7 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4425.0:2023 12.e 12:73912ba8:test-rados-api-smithi170-91706-15::foo:head [setxattr a987 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148) 2023-10-30T17:37:13.331 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:37:13.329+0000 10a5c640 -1 osd.7 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4425.0:2023 12.e 12:73912ba8:test-rados-api-smithi170-91706-15::foo:head [setxattr a987 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148) 2023-10-30T17:37:14.323 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:37:14.320+0000 10a5c640 -1 osd.7 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4425.0:2023 12.e 12:73912ba8:test-rados-api-smithi170-91706-15::foo:head [setxattr a987 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148) 2023-10-30T17:37:15.313 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:37:15.310+0000 10a5c640 -1 osd.7 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4425.0:2023 12.e 12:73912ba8:test-rados-api-smithi170-91706-15::foo:head [setxattr a987 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148) 2023-10-30T17:37:16.311 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:37:16.309+0000 10a5c640 -1 osd.7 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4425.0:2023 12.e 12:73912ba8:test-rados-api-smithi170-91706-15::foo:head [setxattr a987 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148) 2023-10-30T17:37:17.295 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:37:17.294+0000 10a5c640 -1 osd.7 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4425.0:2023 12.e 12:73912ba8:test-rados-api-smithi170-91706-15::foo:head [setxattr a987 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148) 2023-10-30T17:37:17.607 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-30T17:37:17.593+0000 13a62640 -1 bdev(0x16acf370 /var/lib/ceph/osd/ceph-3/block) debug_aio_unlink stalled aio 0x1401cd60 since 2023-10-30T17:36:59.589049+0000, timeout is 5s 2023-10-30T17:37:17.702 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-30T17:37:17.666+0000 1025b640 -1 osd.3 149 get_health_metrics reporting 1 slow ops, oldest is osd_repop(client.4425.0:2023 12.e e149/25 12:73912ba8:test-rados-api-smithi170-91706-15::foo:head v 149'994, mlcod=148'992) 2023-10-30T17:37:19.609 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:06:03.575 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:19.612 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:06:03.578 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:19.641 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:06:03.607 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:19.669 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:03.608 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:19.674 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:03.613 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:19.685 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:06:03.651 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:19.690 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:06:03.656 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:19.700 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:03.640 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:19.721 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:06:03.687 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:28.572 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-10-30T17:37:28.573 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-10-30T17:37:28.573 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-10-30T17:37:28.573 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-10-30T17:37:28.573 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: seed 91883 2023-10-30T17:37:28.573 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-10-30T17:37:28.574 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (15096 ms) 2023-10-30T17:37:28.574 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-10-30T17:37:28.574 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (9902 ms) 2023-10-30T17:37:28.574 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-10-30T17:37:28.575 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (18810 ms) 2023-10-30T17:37:28.575 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-10-30T17:37:28.575 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (7613 ms) 2023-10-30T17:37:28.575 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-10-30T17:37:28.576 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (18158 ms) 2023-10-30T17:37:28.576 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (69579 ms total) 2023-10-30T17:37:28.576 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: 2023-10-30T17:37:28.576 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-10-30T17:37:28.576 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-10-30T17:37:28.576 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (34387 ms) 2023-10-30T17:37:28.577 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-10-30T17:37:28.577 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (9202 ms) 2023-10-30T17:37:28.577 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-10-30T17:37:28.577 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (8817 ms) 2023-10-30T17:37:28.577 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-10-30T17:37:28.577 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (5044 ms) 2023-10-30T17:37:28.578 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-10-30T17:37:28.578 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2733 ms) 2023-10-30T17:37:28.578 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-10-30T17:37:28.578 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6865-gfee4af45/rpm/el9/BUILD/ceph-18.0.0-6865-gfee4af45/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-10-30T17:37:28.579 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: 2023-10-30T17:37:28.579 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-10-30T17:37:28.579 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-10-30T17:37:28.579 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi170-91883-7 2023-10-30T17:37:28.579 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: waiting for snaps to purge 2023-10-30T17:37:28.580 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17605 ms) 2023-10-30T17:37:28.580 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (77789 ms total) 2023-10-30T17:37:28.580 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: 2023-10-30T17:37:28.580 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-10-30T17:37:28.581 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-10-30T17:37:28.581 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3746 ms) 2023-10-30T17:37:28.581 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-10-30T17:37:28.581 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (1952 ms) 2023-10-30T17:37:28.582 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-10-30T17:37:28.582 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2436 ms) 2023-10-30T17:37:28.582 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-10-30T17:37:28.582 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (30430 ms) 2023-10-30T17:37:28.582 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (38564 ms total) 2023-10-30T17:37:28.583 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: 2023-10-30T17:37:28.583 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-10-30T17:37:40.041 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:06:24.007 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:40.048 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:06:24.014 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:40.117 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:06:24.083 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:40.175 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:24.114 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:40.179 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:24.118 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:40.204 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:06:24.170 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:40.215 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:06:24.181 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:40.270 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:24.209 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:40.284 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:06:24.250 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:44.049 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN api_aio: Running main() from gmock_main.cc 2023-10-30T17:37:44.050 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-10-30T17:37:44.050 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [----------] Global test environment set-up. 2023-10-30T17:37:44.051 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-10-30T17:37:44.051 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-10-30T17:37:44.051 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4381 ms) 2023-10-30T17:37:44.052 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-10-30T17:37:44.052 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (25039 ms) 2023-10-30T17:37:44.052 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-10-30T17:37:44.052 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (25329 ms) 2023-10-30T17:37:44.053 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-10-30T17:37:44.053 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (9021 ms) 2023-10-30T17:37:44.053 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-10-30T17:37:44.053 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (6088 ms) 2023-10-30T17:37:44.053 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-10-30T17:37:44.054 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (10465 ms) 2023-10-30T17:37:44.054 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-10-30T17:37:44.054 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (32544 ms) 2023-10-30T17:37:44.054 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-10-30T17:37:44.055 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (28437 ms) 2023-10-30T17:37:44.055 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-10-30T17:37:44.055 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (6358 ms) 2023-10-30T17:37:44.055 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-10-30T17:37:44.056 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (4968 ms) 2023-10-30T17:37:44.056 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-10-30T17:37:44.056 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3915 ms) 2023-10-30T17:37:44.056 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-10-30T17:37:44.056 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3753 ms) 2023-10-30T17:37:44.057 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-10-30T17:37:44.057 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3174 ms) 2023-10-30T17:37:44.057 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-10-30T17:37:44.058 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2856 ms) 2023-10-30T17:37:44.058 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-10-30T17:37:44.058 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.Flush (3151 ms) 2023-10-30T17:37:44.059 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-10-30T17:37:44.059 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2979 ms) 2023-10-30T17:37:44.059 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-10-30T17:37:44.060 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (5061 ms) 2023-10-30T17:37:44.060 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-10-30T17:37:44.060 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4323 ms) 2023-10-30T17:37:44.061 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-10-30T17:37:44.061 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3672 ms) 2023-10-30T17:37:44.061 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-10-30T17:37:44.062 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3343 ms) 2023-10-30T17:37:44.062 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-10-30T17:37:44.062 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (39691 ms) 2023-10-30T17:37:44.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-10-30T17:37:44.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (5193 ms) 2023-10-30T17:37:44.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-10-30T17:37:44.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (4786 ms) 2023-10-30T17:37:44.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-10-30T17:37:44.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4700 ms) 2023-10-30T17:37:44.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-10-30T17:37:44.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3960 ms) 2023-10-30T17:37:44.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-10-30T17:37:44.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3162 ms) 2023-10-30T17:37:44.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [----------] 26 tests from LibRadosAio (250349 ms total) 2023-10-30T17:37:44.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: 2023-10-30T17:37:44.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-10-30T17:37:49.786 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.Simple api_aio_pp: Running main() from gmock_main.cc 2023-10-30T17:37:49.787 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-10-30T17:37:49.787 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-10-30T17:37:49.787 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-10-30T17:37:49.788 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-10-30T17:37:49.788 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4401 ms) 2023-10-30T17:37:49.788 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-10-30T17:37:49.789 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (32218 ms) 2023-10-30T17:37:49.789 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-10-30T17:37:49.790 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (18199 ms) 2023-10-30T17:37:49.790 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-10-30T17:37:49.790 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (5128 ms) 2023-10-30T17:37:49.791 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-10-30T17:37:49.791 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (7316 ms) 2023-10-30T17:37:49.791 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-10-30T17:37:49.792 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (12028 ms) 2023-10-30T17:37:49.792 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-10-30T17:37:49.793 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (33634 ms) 2023-10-30T17:37:49.793 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-10-30T17:37:49.793 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (28372 ms) 2023-10-30T17:37:49.794 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-10-30T17:37:49.794 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (5402 ms) 2023-10-30T17:37:49.794 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-10-30T17:37:49.795 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (4868 ms) 2023-10-30T17:37:49.795 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-10-30T17:37:49.796 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3204 ms) 2023-10-30T17:37:49.796 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-10-30T17:37:49.796 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3850 ms) 2023-10-30T17:37:49.797 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-10-30T17:37:49.797 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3819 ms) 2023-10-30T17:37:49.798 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-10-30T17:37:49.798 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (4961 ms) 2023-10-30T17:37:49.798 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-10-30T17:37:49.799 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (4038 ms) 2023-10-30T17:37:49.799 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-10-30T17:37:49.799 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3202 ms) 2023-10-30T17:37:49.800 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-10-30T17:37:49.800 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2871 ms) 2023-10-30T17:37:49.801 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-10-30T17:37:49.801 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4380 ms) 2023-10-30T17:37:49.801 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-10-30T17:37:49.802 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3621 ms) 2023-10-30T17:37:49.802 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-10-30T17:37:49.803 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3361 ms) 2023-10-30T17:37:49.803 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-10-30T17:37:49.803 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (39674 ms) 2023-10-30T17:37:49.804 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-10-30T17:37:49.804 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (5188 ms) 2023-10-30T17:37:49.804 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-10-30T17:37:49.805 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (4781 ms) 2023-10-30T17:37:49.805 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-10-30T17:37:49.805 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4722 ms) 2023-10-30T17:37:49.806 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-10-30T17:37:49.806 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (4911 ms) 2023-10-30T17:37:49.807 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-10-30T17:37:49.807 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3949 ms) 2023-10-30T17:37:49.807 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-10-30T17:37:50.809 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRad] LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-10-30T17:37:50.809 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (7897 ms) 2023-10-30T17:37:50.809 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-10-30T17:37:50.809 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (6654 ms) 2023-10-30T17:37:50.810 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-10-30T17:37:50.810 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4855 ms) 2023-10-30T17:37:50.810 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (19406 ms total) 2023-10-30T17:37:50.810 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: 2023-10-30T17:37:50.810 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-10-30T17:37:50.811 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (256946 ms total) 2023-10-30T17:37:50.811 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-10-30T17:37:50.811 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-10-30T17:37:50.811 INFO:tasks.workunit.client.0.smithi170.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-10-30T17:37:59.885 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:43.824 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:59.886 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:06:43.852 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:59.888 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:06:43.854 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:59.891 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:43.831 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:59.952 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:43.891 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:59.973 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:06:43.939 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:59.975 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:06:43.941 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:37:59.985 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:06:43.951 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:00.062 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:06:44.028 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:24.939 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:08.905 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:25.002 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:08.941 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:25.009 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:08.975 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:25.010 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:08.949 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:25.085 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:09.051 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:25.093 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:09.032 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:25.098 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:09.064 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:25.192 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:09.158 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:25.197 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:09.163 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:44.459 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-10-30T17:38:44.459 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-10-30T17:38:44.460 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-10-30T17:38:44.460 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-10-30T17:38:44.460 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-10-30T17:38:44.460 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-10-30T17:38:44.460 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-10-30T17:38:44.461 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: 2023-10-30T17:38:44.461 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-10-30T17:38:44.461 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-10-30T17:38:44.461 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (19182 ms) 2023-10-30T17:38:44.461 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-10-30T17:38:44.462 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (22191 ms) 2023-10-30T17:38:44.462 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-10-30T17:38:44.462 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (18195 ms) 2023-10-30T17:38:44.462 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-10-30T17:38:44.462 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (20809 ms) 2023-10-30T17:38:44.463 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-10-30T17:38:44.463 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (40086 ms) 2023-10-30T17:38:44.463 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-10-30T17:38:44.463 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (21689 ms) 2023-10-30T17:38:44.464 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-10-30T17:38:44.464 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (168324 ms) 2023-10-30T17:38:44.464 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (310476 ms total) 2023-10-30T17:38:44.464 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: 2023-10-30T17:38:44.465 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-10-30T17:38:44.465 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (310476 ms total) 2023-10-30T17:38:44.465 INFO:tasks.workunit.client.0.smithi170.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-10-30T17:38:45.205 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:29.171 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:45.294 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:29.259 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:45.299 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:29.265 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:45.333 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:29.272 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:45.339 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:29.305 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:45.341 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:29.280 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:45.387 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:29.353 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:45.403 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:29.342 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:45.403 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:29.369 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:46.277 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:30.216 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:46.280 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:30.219 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:46.351 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:30.290 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:46.418 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:30.384 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:46.419 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:30.385 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:46.437 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:30.403 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:46.445 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:30.411 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:46.446 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:30.412 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:46.469 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:30.435 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:38:50.475 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:b29083e3:::5:head expected=7:b29083e3:::5:head 2023-10-30T17:38:50.475 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:b29083e3:::5:head -> 5 2023-10-30T17:38:50.476 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=5 expected=5 2023-10-30T17:38:50.476 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:863748b0:::15:head 2023-10-30T17:38:50.476 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:863748b0:::15:head expected=7:863748b0:::15:head 2023-10-30T17:38:50.476 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:863748b0:::15:head -> 15 2023-10-30T17:38:50.476 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=15 expected=15 2023-10-30T17:38:50.477 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:6cac518f:::0:head 2023-10-30T17:38:50.477 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:6cac518f:::0:head expected=7:6cac518f:::0:head 2023-10-30T17:38:50.477 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:6cac518f:::0:head -> 0 2023-10-30T17:38:50.477 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=0 expected=0 2023-10-30T17:38:50.477 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:62a1935d:::14:head 2023-10-30T17:38:50.477 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:62a1935d:::14:head expected=7:62a1935d:::14:head 2023-10-30T17:38:50.478 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:62a1935d:::14:head -> 14 2023-10-30T17:38:50.478 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=14 expected=14 2023-10-30T17:38:50.478 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2023-10-30T17:38:50.478 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:5c6b0b28:::7:head expected=7:5c6b0b28:::7:head 2023-10-30T17:38:50.478 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2023-10-30T17:38:50.479 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=7 expected=7 2023-10-30T17:38:50.479 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:02547ec2:::1:head 2023-10-30T17:38:50.479 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:02547ec2:::1:head expected=7:02547ec2:::1:head 2023-10-30T17:38:50.479 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:02547ec2:::1:head -> 1 2023-10-30T17:38:50.479 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=1 expected=1 2023-10-30T17:38:50.480 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:f905c69b:::2:head 2023-10-30T17:38:50.480 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:f905c69b:::2:head expected=7:f905c69b:::2:head 2023-10-30T17:38:50.480 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:f905c69b:::2:head -> 2 2023-10-30T17:38:50.480 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=2 expected=2 2023-10-30T17:38:50.480 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : seek to 7:89d3ae78:::11:head 2023-10-30T17:38:50.481 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : cursor()=7:89d3ae78:::11:head expected=7:89d3ae78:::11:head 2023-10-30T17:38:50.481 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2023-10-30T17:38:50.481 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: : entry=11 expected=11 2023-10-30T17:38:50.481 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (13275 ms) 2023-10-30T17:38:50.481 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-10-30T17:38:50.481 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (150336 ms) 2023-10-30T17:38:50.482 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-10-30T17:38:50.482 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: split 0/5 -> MIN 7:33333333::::head 2023-10-30T17:38:50.482 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: split 1/5 -> 7:33333333::::head 7:66666666::::head 2023-10-30T17:38:50.482 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: split 2/5 -> 7:66666666::::head 7:99999999::::head 2023-10-30T17:38:50.482 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: split 3/5 -> 7:99999999::::head 7:cccccccc::::head 2023-10-30T17:38:50.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: split 4/5 -> 7:cccccccc::::head MAX 2023-10-30T17:38:50.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (56221 ms) 2023-10-30T17:38:50.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [----------] 7 tests from LibRadosList (304548 ms total) 2023-10-30T17:38:50.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 2023-10-30T17:38:50.483 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-10-30T17:38:50.484 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-10-30T17:38:50.484 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2303 ms) 2023-10-30T17:38:50.484 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-10-30T17:38:50.484 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: myset foo1,foo2,foo3 2023-10-30T17:38:50.484 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo1 2023-10-30T17:38:50.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo2 2023-10-30T17:38:50.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo3 2023-10-30T17:38:50.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: myset foo1,foo4,foo5 2023-10-30T17:38:50.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo4 2023-10-30T17:38:50.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo5 2023-10-30T17:38:50.485 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo1 2023-10-30T17:38:50.486 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: myset foo6,foo7 2023-10-30T17:38:50.486 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo7 2023-10-30T17:38:50.486 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: foo6 2023-10-30T17:38:50.486 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-10-30T17:38:50.486 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: ns1:foo4 2023-10-30T17:38:50.487 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: ns1:foo5 2023-10-30T17:38:50.487 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: ns2:foo7 2023-10-30T17:38:50.487 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: ns2:foo6 2023-10-30T17:38:50.487 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: ns1:foo1 2023-10-30T17:38:50.487 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: :foo1 2023-10-30T17:38:50.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: :foo2 2023-10-30T17:38:50.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: :foo3 2023-10-30T17:38:50.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (1480 ms) 2023-10-30T17:38:50.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-10-30T17:38:50.488 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 1 0 2023-10-30T17:38:50.489 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 10 0 2023-10-30T17:38:50.489 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 13 0 2023-10-30T17:38:54.061 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-10-30T17:38:54.061 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-10-30T17:38:54.062 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-10-30T17:38:54.062 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-10-30T17:38:54.062 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-10-30T17:38:54.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-10-30T17:38:54.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: 2023-10-30T17:38:54.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-10-30T17:38:54.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: seed 91706 2023-10-30T17:38:54.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-10-30T17:38:54.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (7 ms) 2023-10-30T17:38:54.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-10-30T17:38:54.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (30807 ms) 2023-10-30T17:38:54.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-10-30T17:38:54.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (7252 ms) 2023-10-30T17:38:54.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-10-30T17:38:54.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (6308 ms) 2023-10-30T17:38:54.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-10-30T17:38:54.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (5190 ms) 2023-10-30T17:38:54.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-10-30T17:38:54.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (1296 ms) 2023-10-30T17:38:54.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-10-30T17:38:54.067 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (1279 ms) 2023-10-30T17:38:54.067 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-10-30T17:38:54.067 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (3916 ms) 2023-10-30T17:38:54.067 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-10-30T17:38:54.067 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2767 ms) 2023-10-30T17:38:54.067 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-10-30T17:38:54.068 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (5839 ms) 2023-10-30T17:38:54.068 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-10-30T17:38:54.068 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2105 ms) 2023-10-30T17:38:54.069 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-10-30T17:38:54.069 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (1120 ms) 2023-10-30T17:38:54.069 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-10-30T17:38:54.069 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (4217 ms) 2023-10-30T17:38:54.069 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-10-30T17:38:54.070 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: osd_max_attr_size = 0 2023-10-30T17:38:54.070 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-10-30T17:38:54.070 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (151485 ms) 2023-10-30T17:38:54.070 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-10-30T17:38:54.070 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (9894 ms) 2023-10-30T17:38:54.071 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-10-30T17:38:54.071 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-10-30T17:38:54.071 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: done waiting, doing copies 2023-10-30T17:38:54.071 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: waiting for final deep scrubs... 2023-10-30T17:38:54.071 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: done waiting 2023-10-30T17:38:54.072 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (75966 ms) 2023-10-30T17:38:54.072 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-10-30T17:38:54.072 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (310 ms) 2023-10-30T17:38:54.073 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-10-30T17:38:54.073 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (44 ms) 2023-10-30T17:38:54.073 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-10-30T17:38:54.074 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5117 ms) 2023-10-30T17:38:54.074 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-10-30T17:38:54.074 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-10-30T17:38:54.074 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-10-30T17:38:54.074 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-10-30T17:38:54.075 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-10-30T17:38:54.075 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-10-30T17:38:58.384 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [--- api_list: 7 0 2023-10-30T17:38:58.385 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 14 0 2023-10-30T17:38:58.385 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 0 0 2023-10-30T17:38:58.385 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 15 0 2023-10-30T17:38:58.385 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 11 0 2023-10-30T17:38:58.386 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 5 0 2023-10-30T17:38:58.386 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 8 0 2023-10-30T17:38:58.386 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 6 0 2023-10-30T17:38:58.386 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 3 0 2023-10-30T17:38:58.386 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 4 0 2023-10-30T17:38:58.387 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 12 0 2023-10-30T17:38:58.387 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 9 0 2023-10-30T17:38:58.387 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 2 0 2023-10-30T17:38:58.387 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-10-30T17:38:58.387 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (3535 ms) 2023-10-30T17:38:58.387 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [----------] 3 tests from LibRadosListEC (7318 ms total) 2023-10-30T17:38:58.388 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 2023-10-30T17:38:58.388 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-10-30T17:38:58.388 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-10-30T17:38:58.388 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3234 ms) 2023-10-30T17:38:58.388 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [----------] 1 test from LibRadosListNP (3234 ms total) 2023-10-30T17:38:58.389 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: 2023-10-30T17:38:58.389 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [----------] Global test environment tear-down 2023-10-30T17:38:58.389 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [==========] 11 tests from 3 test suites ran. (324630 ms total) 2023-10-30T17:38:58.389 INFO:tasks.workunit.client.0.smithi170.stdout: api_list: [ PASSED ] 11 tests. 2023-10-30T17:39:05.359 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:49.325 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:05.413 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:49.352 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:05.416 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:49.382 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:05.421 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:49.360 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:05.469 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:49.408 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:05.493 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:07:49.459 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:05.542 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:49.508 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:05.546 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:49.512 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:05.600 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:07:49.566 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:23.898 INFO:tasks.workunit.client.0.smithi170.stdout:-------] 22 tests from LibRadosMiscPP (314920 ms total) 2023-10-30T17:39:23.898 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: 2023-10-30T17:39:23.898 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-10-30T17:39:23.898 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-10-30T17:39:23.899 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2794 ms) 2023-10-30T17:39:23.899 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2794 ms total) 2023-10-30T17:39:23.899 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: 2023-10-30T17:39:23.899 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-10-30T17:39:23.899 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-10-30T17:39:23.900 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (192 ms) 2023-10-30T17:39:23.900 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-10-30T17:39:23.900 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (427 ms) 2023-10-30T17:39:23.900 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (619 ms total) 2023-10-30T17:39:23.900 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: 2023-10-30T17:39:23.901 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-10-30T17:39:23.901 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-10-30T17:39:23.901 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (723 ms) 2023-10-30T17:39:23.901 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-10-30T17:39:23.901 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (45 ms) 2023-10-30T17:39:23.902 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (768 ms total) 2023-10-30T17:39:23.902 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: 2023-10-30T17:39:23.902 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-10-30T17:39:23.902 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-10-30T17:39:23.902 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (426 ms) 2023-10-30T17:39:23.903 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-10-30T17:39:23.903 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (254 ms) 2023-10-30T17:39:23.903 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (680 ms total) 2023-10-30T17:39:23.903 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: 2023-10-30T17:39:23.903 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-10-30T17:39:23.904 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-10-30T17:39:23.904 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1498 ms) 2023-10-30T17:39:23.904 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1498 ms total) 2023-10-30T17:39:23.904 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: 2023-10-30T17:39:23.904 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-10-30T17:39:23.904 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (350096 ms total) 2023-10-30T17:39:23.905 INFO:tasks.workunit.client.0.smithi170.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-10-30T17:39:25.423 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:08:09.389 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:25.473 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:08:09.439 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:25.520 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:09.459 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:25.521 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:09.460 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:25.521 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:08:09.487 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:25.522 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:08:09.489 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:25.535 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:08:09.501 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:25.563 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:09.502 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:25.567 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:08:09.533 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:43.887 INFO:tasks.workunit.client.0.smithi170.stdout:Write 2023-10-30T17:39:43.888 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7754 ms) 2023-10-30T17:39:43.888 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-10-30T17:39:43.888 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7454 ms) 2023-10-30T17:39:43.889 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-10-30T17:39:43.889 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7092 ms) 2023-10-30T17:39:43.889 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-10-30T17:39:43.889 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7965 ms) 2023-10-30T17:39:43.889 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-10-30T17:39:43.890 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7705 ms) 2023-10-30T17:39:43.890 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-10-30T17:39:43.890 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7292 ms) 2023-10-30T17:39:43.890 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-10-30T17:39:43.890 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7680 ms) 2023-10-30T17:39:43.891 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-10-30T17:39:43.891 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7457 ms) 2023-10-30T17:39:43.891 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-10-30T17:39:43.891 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7466 ms) 2023-10-30T17:39:43.891 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-10-30T17:39:43.892 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7525 ms) 2023-10-30T17:39:43.892 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-10-30T17:39:43.892 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7097 ms) 2023-10-30T17:39:43.892 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-10-30T17:39:43.892 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7252 ms) 2023-10-30T17:39:43.892 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-10-30T17:39:43.893 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6928 ms) 2023-10-30T17:39:43.893 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-10-30T17:39:43.893 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7694 ms) 2023-10-30T17:39:43.893 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-10-30T17:39:43.893 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7108 ms) 2023-10-30T17:39:43.894 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-10-30T17:39:43.894 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (8367 ms) 2023-10-30T17:39:43.894 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (119836 ms total) 2023-10-30T17:39:43.894 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: 2023-10-30T17:39:43.894 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [----------] Global test environment tear-down 2023-10-30T17:39:43.895 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (370185 ms total) 2023-10-30T17:39:43.895 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio: [ PASSED ] 42 tests. 2023-10-30T17:39:45.456 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:08:29.422 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:45.508 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:29.447 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:45.510 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:08:29.476 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:45.513 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:08:29.479 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:45.514 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:08:29.480 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:45.515 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:29.453 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:45.556 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:08:29.522 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:45.562 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:29.501 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:39:45.565 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:08:29.531 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:05.595 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:08:49.561 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:05.601 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:08:49.567 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:05.651 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:08:49.617 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:05.653 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:08:49.619 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:05.655 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:08:49.621 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:05.667 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:49.606 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:05.670 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:49.609 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:05.699 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:08:49.665 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:05.709 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:49.648 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:20.785 INFO:tasks.workunit.client.0.smithi170.stdout:osAio.RoundTripAppendPP (3980 ms) 2023-10-30T17:40:20.785 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-10-30T17:40:20.786 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3033 ms) 2023-10-30T17:40:20.786 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-10-30T17:40:20.786 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3085 ms) 2023-10-30T17:40:20.786 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-10-30T17:40:20.787 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3422 ms) 2023-10-30T17:40:20.787 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-10-30T17:40:20.787 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: setting pool EIO 2023-10-30T17:40:20.788 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: max_success 0, min_failed 1 2023-10-30T17:40:20.788 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4929 ms) 2023-10-30T17:40:20.788 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-10-30T17:40:20.788 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4132 ms) 2023-10-30T17:40:20.789 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (274680 ms total) 2023-10-30T17:40:20.789 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: 2023-10-30T17:40:20.789 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-10-30T17:40:20.789 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-10-30T17:40:20.790 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3490 ms) 2023-10-30T17:40:20.790 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-10-30T17:40:20.790 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (11165 ms) 2023-10-30T17:40:20.790 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-10-30T17:40:20.791 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15102 ms) 2023-10-30T17:40:20.791 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-10-30T17:40:20.791 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3231 ms) 2023-10-30T17:40:20.791 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (32989 ms total) 2023-10-30T17:40:20.791 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: 2023-10-30T17:40:20.792 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-10-30T17:40:20.792 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-10-30T17:40:20.792 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3090 ms) 2023-10-30T17:40:20.792 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3090 ms total) 2023-10-30T17:40:20.792 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: 2023-10-30T17:40:20.793 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-10-30T17:40:20.793 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-10-30T17:40:20.793 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14962 ms) 2023-10-30T17:40:20.793 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-10-30T17:40:20.793 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7037 ms) 2023-10-30T17:40:20.794 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-10-30T17:40:20.794 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7316 ms) 2023-10-30T17:40:20.794 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-10-30T17:40:20.794 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6925 ms) 2023-10-30T17:40:20.794 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-10-30T17:40:20.795 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3192 ms) 2023-10-30T17:40:20.795 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-10-30T17:40:20.795 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7625 ms) 2023-10-30T17:40:20.795 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-10-30T17:40:20.795 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7146 ms) 2023-10-30T17:40:20.795 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-10-30T17:40:20.796 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6777 ms) 2023-10-30T17:40:20.796 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-10-30T17:40:20.796 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7306 ms) 2023-10-30T17:40:20.796 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-10-30T17:40:20.796 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6975 ms) 2023-10-30T17:40:20.797 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-10-30T17:40:20.797 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6857 ms) 2023-10-30T17:40:20.797 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-10-30T17:40:20.797 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7079 ms) 2023-10-30T17:40:20.797 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-10-30T17:40:20.798 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7122 ms) 2023-10-30T17:40:25.672 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:09.637 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:25.673 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:09.639 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:25.715 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:09.681 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:25.719 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:09.658 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:25.720 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:09.660 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:25.729 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:09.695 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:25.732 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:09.698 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:25.761 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:09.700 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:25.776 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:09.741 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:31.049 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:15.014 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:31.103 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:15.069 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:31.118 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:15.057 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:31.122 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:15.061 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:31.139 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:15.104 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:31.146 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:15.085 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:31.186 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:15.150 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:31.189 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:15.155 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:31.215 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:15.181 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:45.898 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:29.864 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:45.902 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:29.869 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:45.934 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:29.900 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:45.946 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:29.885 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:45.947 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:29.886 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:45.949 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:29.916 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:45.958 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:29.924 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:45.977 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:29.916 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:40:46.003 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:29.969 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:05.968 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:49.934 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:06.019 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:49.985 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:06.064 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:09:50.030 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:06.074 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:50.013 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:06.074 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:50.041 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:06.075 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:50.014 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:06.076 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:50.042 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:06.116 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:50.055 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:06.116 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:09:50.082 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:08.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-10-30T17:41:08.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3330 ms) 2023-10-30T17:41:08.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-10-30T17:41:08.063 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6857 ms) 2023-10-30T17:41:08.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-10-30T17:41:08.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6840 ms) 2023-10-30T17:41:08.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-10-30T17:41:08.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (8019 ms) 2023-10-30T17:41:08.064 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-10-30T17:41:08.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7697 ms) 2023-10-30T17:41:08.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-10-30T17:41:08.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6759 ms) 2023-10-30T17:41:08.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-10-30T17:41:08.065 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7774 ms) 2023-10-30T17:41:08.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (143595 ms total) 2023-10-30T17:41:08.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: 2023-10-30T17:41:08.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-10-30T17:41:08.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (454354 ms total) 2023-10-30T17:41:08.066 INFO:tasks.workunit.client.0.smithi170.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-10-30T17:41:25.854 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:10:09.820 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:25.901 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:09.840 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:25.903 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:09.842 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:25.904 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:10:09.871 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:25.926 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:09.865 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:25.931 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:10:09.897 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:25.970 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:10:09.936 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:25.972 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:10:09.938 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:25.991 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:10:09.957 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:51.100 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:10:35.066 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:51.103 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:10:35.069 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:51.138 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:10:35.104 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:51.152 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:10:35.118 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:51.153 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:10:35.119 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:51.162 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:35.101 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:51.163 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:35.102 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:51.188 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:10:35.154 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:41:51.198 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:35.136 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:11.035 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:10:55.001 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:11.036 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:54.975 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:11.038 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:10:55.004 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:11.039 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:54.978 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:11.071 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:55.010 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:11.072 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:10:55.038 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:11.084 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:10:55.050 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:11.088 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:10:55.054 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:11.123 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:10:55.089 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:31.197 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:15.163 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:31.199 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:15.165 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:31.232 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:15.198 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:31.246 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:15.185 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:31.248 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:15.187 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:31.254 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:15.221 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:31.260 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:15.226 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:31.284 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:15.223 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:31.298 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:15.264 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:32.248 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-10-30T17:42:32.248 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-10-30T17:42:32.248 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-10-30T17:42:32.249 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: seed 91745 2023-10-30T17:42:32.249 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-10-30T17:42:32.249 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (67031 ms) 2023-10-30T17:42:32.249 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-10-30T17:42:32.249 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (71986 ms) 2023-10-30T17:42:32.250 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-10-30T17:42:32.250 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (175 ms) 2023-10-30T17:42:32.250 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (139192 ms total) 2023-10-30T17:42:32.251 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: 2023-10-30T17:42:32.251 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-10-30T17:42:32.251 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-10-30T17:42:32.251 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (12000 ms) 2023-10-30T17:42:32.251 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-10-30T17:42:32.252 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (14920 ms) 2023-10-30T17:42:32.252 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-10-30T17:42:32.252 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (55014 ms) 2023-10-30T17:42:32.252 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-10-30T17:42:32.252 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: my_snaps [4] 2023-10-30T17:42:32.253 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: my_snaps [5,4] 2023-10-30T17:42:32.253 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: my_snaps [6,5,4] 2023-10-30T17:42:32.253 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-10-30T17:42:32.253 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: promoting some heads 2023-10-30T17:42:32.253 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: promoting from clones for snap 7 2023-10-30T17:42:32.253 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: promoting from clones for snap 6 2023-10-30T17:42:32.254 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: promoting from clones for snap 5 2023-10-30T17:42:32.254 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: promoting from clones for snap 4 2023-10-30T17:42:32.254 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: waiting for scrubs... 2023-10-30T17:42:32.254 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: done waiting 2023-10-30T17:42:32.255 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (149378 ms) 2023-10-30T17:42:32.255 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-10-30T17:42:32.255 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10228 ms) 2023-10-30T17:42:32.256 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-10-30T17:42:32.256 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (9689 ms) 2023-10-30T17:42:32.256 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-10-30T17:42:32.256 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8088 ms) 2023-10-30T17:42:32.256 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-10-30T17:42:32.256 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10215 ms) 2023-10-30T17:42:32.257 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-10-30T17:42:32.257 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10234 ms) 2023-10-30T17:42:32.257 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-10-30T17:42:32.258 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9993 ms) 2023-10-30T17:42:32.258 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-10-30T17:42:32.258 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10204 ms) 2023-10-30T17:42:32.258 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-10-30T17:42:32.258 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (39309 ms) 2023-10-30T17:42:32.259 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-10-30T17:42:32.259 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8747 ms) 2023-10-30T17:42:32.259 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-10-30T17:42:32.259 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (10204 ms) 2023-10-30T17:42:32.259 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-10-30T17:42:32.260 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (14679 ms) 2023-10-30T17:42:32.260 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-10-30T17:42:32.260 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7891 ms) 2023-10-30T17:42:32.260 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-10-30T17:42:32.260 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11135 ms) 2023-10-30T17:42:34.557 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:18.523 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:34.559 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:18.525 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:34.571 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:18.537 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:34.578 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:18.544 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:34.621 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:18.560 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:34.630 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:18.569 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:34.663 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:18.629 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:34.675 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:18.614 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:34.691 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:18.657 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:51.273 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:35.238 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:51.276 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:35.241 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:51.303 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:35.269 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:51.324 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:35.262 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:51.325 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:35.264 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:51.325 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:35.291 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:51.326 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:35.292 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:51.354 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:35.293 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:51.354 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:35.320 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:42:53.964 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-10-30T17:42:53.965 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: hmm, no HitSet yet 2023-10-30T17:42:53.965 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: ok, hit_set contains 200:602f83fe:::foo:head 2023-10-30T17:42:53.965 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9915 ms) 2023-10-30T17:42:53.965 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-10-30T17:42:53.965 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg_num = 32 2023-10-30T17:42:53.966 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 0 ls 1698687769,0 2023-10-30T17:42:53.966 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 1 ls 1698687769,0 2023-10-30T17:42:53.966 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 2 ls 1698687769,0 2023-10-30T17:42:53.966 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 3 ls 1698687769,0 2023-10-30T17:42:53.966 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 4 ls 1698687769,0 2023-10-30T17:42:53.967 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 5 ls 1698687769,0 2023-10-30T17:42:53.967 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 6 ls 1698687769,0 2023-10-30T17:42:53.967 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 7 ls 1698687769,0 2023-10-30T17:42:53.967 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 8 ls 1698687769,0 2023-10-30T17:42:53.967 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 9 ls 1698687769,0 2023-10-30T17:42:53.968 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 10 ls 1698687769,0 2023-10-30T17:42:53.968 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 11 ls 1698687769,0 2023-10-30T17:42:53.968 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 12 ls 1698687769,0 2023-10-30T17:42:53.968 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 13 ls 1698687769,0 2023-10-30T17:42:53.969 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 14 ls 1698687769,0 2023-10-30T17:42:53.969 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 15 ls 1698687769,0 2023-10-30T17:42:53.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 16 ls 1698687769,0 2023-10-30T17:42:53.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 17 ls 1698687769,0 2023-10-30T17:42:53.970 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 18 ls 1698687769,0 2023-10-30T17:42:53.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 19 ls 1698687769,0 2023-10-30T17:42:53.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 20 ls 1698687769,0 2023-10-30T17:42:53.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 21 ls 1698687769,0 2023-10-30T17:42:53.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 22 ls 1698687769,0 2023-10-30T17:42:53.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 23 ls 1698687769,0 2023-10-30T17:42:53.971 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 24 ls 1698687769,0 2023-10-30T17:42:53.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 25 ls 1698687769,0 2023-10-30T17:42:53.972 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 26 ls 1698687769,0 2023-10-30T17:42:53.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 27 ls 1698687769,0 2023-10-30T17:42:53.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 28 ls 1698687769,0 2023-10-30T17:42:53.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 29 ls 1698687769,0 2023-10-30T17:42:53.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 30 ls 1698687769,0 2023-10-30T17:42:53.973 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg 31 ls 1698687769,0 2023-10-30T17:42:53.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: pg_num = 32 2023-10-30T17:42:53.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:6cac518f:::0:head 2023-10-30T17:42:53.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:02547ec2:::1:head 2023-10-30T17:42:53.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:f905c69b:::2:head 2023-10-30T17:42:53.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:cfc208b3:::3:head 2023-10-30T17:42:53.974 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d83876eb:::4:head 2023-10-30T17:42:53.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:b29083e3:::5:head 2023-10-30T17:42:53.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c4fdafeb:::6:head 2023-10-30T17:42:53.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:5c6b0b28:::7:head 2023-10-30T17:42:53.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:bd63b0f1:::8:head 2023-10-30T17:42:53.975 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:e960b815:::9:head 2023-10-30T17:42:53.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:52ea6a34:::10:head 2023-10-30T17:42:53.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:89d3ae78:::11:head 2023-10-30T17:42:53.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:de5d7c5f:::12:head 2023-10-30T17:42:53.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:566253c9:::13:head 2023-10-30T17:42:53.976 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:62a1935d:::14:head 2023-10-30T17:42:53.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:863748b0:::15:head 2023-10-30T17:42:53.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:3958e169:::16:head 2023-10-30T17:42:53.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:4d4dabf9:::17:head 2023-10-30T17:42:53.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:8391935d:::18:head 2023-10-30T17:42:53.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:28883081:::19:head 2023-10-30T17:42:53.977 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:69259c59:::20:head 2023-10-30T17:42:53.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:4bdb80b7:::21:head 2023-10-30T17:42:53.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:a11c5d71:::22:head 2023-10-30T17:42:53.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:271af37b:::23:head 2023-10-30T17:42:53.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:95b121be:::24:head 2023-10-30T17:42:53.978 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:58d1031b:::25:head 2023-10-30T17:42:53.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:0a050783:::26:head 2023-10-30T17:42:53.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c709704c:::27:head 2023-10-30T17:42:53.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:cbe56eaf:::28:head 2023-10-30T17:42:53.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:86b4b162:::29:head 2023-10-30T17:42:53.979 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:70d89383:::30:head 2023-10-30T17:42:53.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:dd450c7c:::31:head 2023-10-30T17:42:53.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:6d5729b1:::32:head 2023-10-30T17:42:53.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c388f3fb:::33:head 2023-10-30T17:42:53.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:56cfea31:::34:head 2023-10-30T17:42:53.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:9dbc1bf7:::35:head 2023-10-30T17:42:53.980 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:40b74ccd:::36:head 2023-10-30T17:42:53.981 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:4d5aaf42:::37:head 2023-10-30T17:42:53.981 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:920f362c:::38:head 2023-10-30T17:42:53.981 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:6cc53222:::39:head 2023-10-30T17:42:53.981 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:9cad833f:::40:head 2023-10-30T17:42:53.981 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:1ea84d41:::41:head 2023-10-30T17:42:53.982 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c4480ef6:::42:head 2023-10-30T17:42:53.982 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:a694361e:::43:head 2023-10-30T17:42:53.982 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d1bd33e9:::44:head 2023-10-30T17:42:53.982 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:ddc2cd5d:::45:head 2023-10-30T17:42:53.982 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:2b782207:::46:head 2023-10-30T17:42:53.982 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:7b187fca:::47:head 2023-10-30T17:42:53.983 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:90ecdf6f:::48:head 2023-10-30T17:42:53.983 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:a5ed95fe:::49:head 2023-10-30T17:42:53.983 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:ea0eaa55:::50:head 2023-10-30T17:42:53.983 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:f33ef17b:::51:head 2023-10-30T17:42:53.983 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:a0d1b2f6:::52:head 2023-10-30T17:42:53.984 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:60c5229e:::53:head 2023-10-30T17:42:53.984 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:edcbc575:::54:head 2023-10-30T17:42:53.984 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:102cf253:::55:head 2023-10-30T17:42:53.984 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:efb7fb0b:::56:head 2023-10-30T17:42:53.984 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:50d0a326:::57:head 2023-10-30T17:42:53.985 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d4dc5daf:::58:head 2023-10-30T17:42:53.985 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:3a130462:::59:head 2023-10-30T17:42:53.985 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:ec87ed71:::60:head 2023-10-30T17:42:53.985 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d5bc9454:::61:head 2023-10-30T17:42:53.985 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:3ddfe313:::62:head 2023-10-30T17:42:53.985 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:7c2816b9:::63:head 2023-10-30T17:42:53.986 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:47e00e4d:::64:head 2023-10-30T17:42:53.986 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c6410c18:::65:head 2023-10-30T17:42:53.986 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:b48ed237:::66:head 2023-10-30T17:42:53.986 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:cd63ad31:::67:head 2023-10-30T17:42:53.986 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:b179e92b:::68:head 2023-10-30T17:42:53.987 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:0d9f741a:::69:head 2023-10-30T17:42:53.987 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:6d3352ae:::70:head 2023-10-30T17:42:53.987 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c6d5c19e:::71:head 2023-10-30T17:42:53.987 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:bc4729c3:::72:head 2023-10-30T17:42:53.987 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:77e930b9:::73:head 2023-10-30T17:42:53.987 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:0abeecfd:::74:head 2023-10-30T17:42:53.988 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:b7c37e15:::75:head 2023-10-30T17:42:53.988 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:b6378398:::76:head 2023-10-30T17:42:53.988 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:02bd68de:::77:head 2023-10-30T17:42:53.988 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:cc795d2d:::78:head 2023-10-30T17:42:53.988 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:630d4fea:::79:head 2023-10-30T17:42:53.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:e0d29ef5:::80:head 2023-10-30T17:42:53.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:fd6f13d2:::81:head 2023-10-30T17:42:53.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:606461d5:::82:head 2023-10-30T17:42:53.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:eadbdc43:::83:head 2023-10-30T17:42:53.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:8761d0bb:::84:head 2023-10-30T17:42:53.989 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:9ef0186f:::85:head 2023-10-30T17:42:53.990 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:e0d41294:::86:head 2023-10-30T17:42:53.990 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:961de695:::87:head 2023-10-30T17:42:53.990 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:1423148f:::88:head 2023-10-30T17:42:53.990 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:633a8fa2:::89:head 2023-10-30T17:42:53.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:a8653809:::90:head 2023-10-30T17:42:53.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:3dac8b33:::91:head 2023-10-30T17:42:53.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:35aad435:::92:head 2023-10-30T17:42:53.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:f6dcc343:::93:head 2023-10-30T17:42:53.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:dbbdad87:::94:head 2023-10-30T17:42:53.991 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:1cb48ce0:::95:head 2023-10-30T17:42:53.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:03cd461c:::96:head 2023-10-30T17:42:53.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:17a4ea99:::97:head 2023-10-30T17:42:53.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:9993c9a7:::98:head 2023-10-30T17:42:53.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:6394211c:::99:head 2023-10-30T17:42:53.992 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:94c7ae57:::100:head 2023-10-30T17:42:53.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:6fdee5bb:::101:head 2023-10-30T17:42:53.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:9a477fd1:::102:head 2023-10-30T17:42:53.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:eb850916:::103:head 2023-10-30T17:42:53.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:affc56b9:::104:head 2023-10-30T17:42:53.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:b42dc814:::105:head 2023-10-30T17:42:53.993 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:f319f8f0:::106:head 2023-10-30T17:42:53.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:9a40b9de:::107:head 2023-10-30T17:42:53.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:8b524f28:::108:head 2023-10-30T17:42:53.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:e3de589f:::109:head 2023-10-30T17:42:53.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:90f90a5b:::110:head 2023-10-30T17:42:53.994 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:a7b4f1d7:::111:head 2023-10-30T17:42:53.995 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:af51766e:::112:head 2023-10-30T17:42:53.995 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:b6f90bd1:::113:head 2023-10-30T17:42:53.995 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:e0261208:::114:head 2023-10-30T17:42:53.995 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c9569ef7:::115:head 2023-10-30T17:42:53.995 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:61bebe50:::116:head 2023-10-30T17:42:53.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:fe93412b:::117:head 2023-10-30T17:42:53.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d3d38bee:::118:head 2023-10-30T17:42:53.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:3100ba0c:::119:head 2023-10-30T17:42:53.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d0560ada:::120:head 2023-10-30T17:42:53.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:f0ea8b35:::121:head 2023-10-30T17:42:53.996 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:766f231a:::122:head 2023-10-30T17:42:53.997 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:a07a2582:::123:head 2023-10-30T17:42:53.997 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:bd7c6b3a:::124:head 2023-10-30T17:42:53.997 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:fb2ddaff:::125:head 2023-10-30T17:42:53.997 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:4408e1fe:::126:head 2023-10-30T17:42:53.997 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:ee1df7a7:::127:head 2023-10-30T17:42:53.998 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c3002909:::128:head 2023-10-30T17:42:53.998 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:4f48ffa9:::129:head 2023-10-30T17:42:53.998 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:edf38733:::130:head 2023-10-30T17:42:53.998 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c08425c0:::131:head 2023-10-30T17:42:53.998 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:5f902d98:::132:head 2023-10-30T17:42:53.998 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:41ea2c93:::133:head 2023-10-30T17:42:53.999 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:813cee13:::134:head 2023-10-30T17:42:53.999 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:0131818d:::135:head 2023-10-30T17:42:53.999 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:26ba5a85:::136:head 2023-10-30T17:42:53.999 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:381b8a5a:::137:head 2023-10-30T17:42:53.999 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:28797e47:::138:head 2023-10-30T17:42:54.000 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:bfca7f22:::139:head 2023-10-30T17:42:54.000 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:36807075:::140:head 2023-10-30T17:42:54.000 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:80b03975:::141:head 2023-10-30T17:42:54.000 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:5c15709b:::142:head 2023-10-30T17:42:54.000 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:f39ea15e:::143:head 2023-10-30T17:42:54.001 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:ea992956:::144:head 2023-10-30T17:42:54.001 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:48887b1c:::145:head 2023-10-30T17:42:54.001 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:9f24a9dd:::146:head 2023-10-30T17:42:54.001 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:987f100b:::147:head 2023-10-30T17:42:54.001 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d2dd3581:::148:head 2023-10-30T17:42:54.002 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:7fed1808:::149:head 2023-10-30T17:42:54.002 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c80b70e9:::150:head 2023-10-30T17:42:54.002 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:85ed90f9:::151:head 2023-10-30T17:42:54.002 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:36428b24:::152:head 2023-10-30T17:42:54.002 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d044c34a:::153:head 2023-10-30T17:42:54.003 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:7c18bf58:::154:head 2023-10-30T17:42:54.003 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d1c21232:::155:head 2023-10-30T17:42:54.003 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:a7a3c575:::156:head 2023-10-30T17:42:54.003 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:87da0633:::157:head 2023-10-30T17:42:54.003 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d5ac3822:::158:head 2023-10-30T17:42:54.003 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:3f20522d:::159:head 2023-10-30T17:42:54.004 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:6ca26563:::160:head 2023-10-30T17:42:54.004 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:532ce135:::161:head 2023-10-30T17:42:54.004 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:c78863e6:::162:head 2023-10-30T17:43:11.207 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:55.173 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:11.253 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:55.219 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:11.282 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:11:55.248 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:11.299 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:55.238 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:11.301 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:55.241 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:11.303 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:55.269 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:11.306 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:55.272 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:11.329 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:55.268 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:11.335 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:11:55.302 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:26.482 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:10.448 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:26.540 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:10.479 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:26.544 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:10.510 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:26.564 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:10.503 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:26.584 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:10.523 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:26.600 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:10.566 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:26.608 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:10.575 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:26.610 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:10.576 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:26.628 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:10.594 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:31.343 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:15.309 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:31.389 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:15.355 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:31.406 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:15.372 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:31.437 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:15.403 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:31.438 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:15.377 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:31.439 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:15.379 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:31.440 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:15.406 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:31.452 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:15.391 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:31.455 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:15.422 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:41.439 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:25.405 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:41.441 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:25.408 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:41.464 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:25.430 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:41.484 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:25.450 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:41.487 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:25.426 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:41.488 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:25.454 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:41.489 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:25.428 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:41.509 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:25.449 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:41.520 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:25.486 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:51.459 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:35.425 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:51.463 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:35.429 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:51.483 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:35.449 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:51.508 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:35.474 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:51.509 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:35.448 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:51.511 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:35.477 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:51.512 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:35.451 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:51.527 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:35.493 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:51.530 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:35.469 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:56.428 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:40.394 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:56.474 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:40.440 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:56.487 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:40.453 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:56.515 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:40.454 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:56.519 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:40.459 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:56.523 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:40.489 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:56.526 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:40.492 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:56.531 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:40.470 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:43:56.536 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:40.503 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:00.222 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:7462ddf6:::163:head 2023-10-30T17:44:00.222 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:5d165639:::164:head 2023-10-30T17:44:00.222 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:f43765fc:::165:head 2023-10-30T17:44:00.223 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:b4c720e9:::166:head 2023-10-30T17:44:00.223 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:e694b040:::167:head 2023-10-30T17:44:00.223 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:afa38db2:::168:head 2023-10-30T17:44:00.223 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:77ba9f53:::169:head 2023-10-30T17:44:00.223 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:87495034:::170:head 2023-10-30T17:44:00.224 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:7c96bf0e:::171:head 2023-10-30T17:44:00.224 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:dbe346cc:::172:head 2023-10-30T17:44:00.224 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:e943ec24:::173:head 2023-10-30T17:44:00.224 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:f97a9c0c:::174:head 2023-10-30T17:44:00.224 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:6f26e74d:::175:head 2023-10-30T17:44:00.225 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:4f95e106:::176:head 2023-10-30T17:44:00.225 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:0e6f2f8f:::177:head 2023-10-30T17:44:00.225 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:05db05f1:::178:head 2023-10-30T17:44:00.225 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:38a78d66:::179:head 2023-10-30T17:44:00.225 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:d095610b:::180:head 2023-10-30T17:44:00.226 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:a1a9d709:::181:head 2023-10-30T17:44:00.226 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:1e5d39db:::182:head 2023-10-30T17:44:00.226 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:f7df4fb9:::183:head 2023-10-30T17:44:00.226 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:03a7f161:::184:head 2023-10-30T17:44:00.226 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:ba70721e:::185:head 2023-10-30T17:44:00.227 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:28e5662d:::186:head 2023-10-30T17:44:00.227 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:973d52de:::187:head 2023-10-30T17:44:00.227 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:4303eb1c:::188:head 2023-10-30T17:44:00.227 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:b990b48e:::189:head 2023-10-30T17:44:00.227 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:29b8165b:::190:head 2023-10-30T17:44:00.228 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:3547f197:::191:head 2023-10-30T17:44:00.228 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:7e260936:::192:head 2023-10-30T17:44:00.228 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:1abec7b1:::193:head 2023-10-30T17:44:00.228 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:10fdda93:::194:head 2023-10-30T17:44:00.228 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:15817eea:::195:head 2023-10-30T17:44:00.228 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:770bab57:::196:head 2023-10-30T17:44:00.229 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:ed9e13e7:::197:head 2023-10-30T17:44:00.229 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:71471a8f:::198:head 2023-10-30T17:44:00.229 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: checking for 201:10fb1d02:::199:head 2023-10-30T17:44:00.229 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21307 ms) 2023-10-30T17:44:00.229 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-10-30T17:44:00.230 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,0 2023-10-30T17:44:00.230 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: first is 1698687791 2023-10-30T17:44:00.230 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,0 2023-10-30T17:44:00.230 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,0 2023-10-30T17:44:00.230 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,1698687793,1698687794,0 2023-10-30T17:44:00.231 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,1698687793,1698687794,0 2023-10-30T17:44:00.231 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,1698687793,1698687794,0 2023-10-30T17:44:00.231 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,1698687793,1698687794,1698687796,1698687797,0 2023-10-30T17:44:00.231 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,1698687793,1698687794,1698687796,1698687797,0 2023-10-30T17:44:00.231 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,1698687793,1698687794,1698687796,1698687797,0 2023-10-30T17:44:00.231 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,1698687793,1698687794,1698687796,1698687797,1698687799,1698687800,0 2023-10-30T17:44:00.232 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,1698687793,1698687794,1698687796,1698687797,1698687799,1698687800,0 2023-10-30T17:44:00.232 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687791,1698687793,1698687794,1698687796,1698687797,1698687799,1698687800,0 2023-10-30T17:44:00.232 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698687794,1698687796,1698687797,1698687799,1698687800,1698687802,1698687803,0 2023-10-30T17:44:00.232 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: first now 1698687794, trimmed 2023-10-30T17:44:00.232 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21616 ms) 2023-10-30T17:44:00.233 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-10-30T17:44:00.233 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: foo0 2023-10-30T17:44:00.233 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-10-30T17:44:00.233 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14552 ms) 2023-10-30T17:44:00.233 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-10-30T17:44:11.520 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:55.459 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:11.521 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:55.487 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:11.521 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:55.460 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:11.523 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:55.489 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:11.524 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:55.490 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:11.526 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:55.492 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:11.550 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:55.489 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:11.552 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:55.518 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:11.564 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:55.530 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:12.368 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:56.307 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:12.369 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:56.308 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:12.379 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:56.318 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:12.443 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:56.409 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:12.443 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:56.410 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:12.454 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:12:56.419 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:12.465 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:56.431 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:12.466 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:56.432 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:12.476 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:12:56.442 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:26.637 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:10.603 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:26.688 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:10.653 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:26.693 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:10.632 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:26.694 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:10.633 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:26.709 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:10.675 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:26.718 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:10.684 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:26.720 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:10.686 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:26.727 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:10.666 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:26.749 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:10.715 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:31.544 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:15.510 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:31.592 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:15.558 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:31.613 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:15.579 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:31.633 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:15.572 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:31.637 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:15.576 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:31.645 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:15.611 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:31.648 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:15.614 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:31.652 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:15.590 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:31.663 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:15.628 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:41.711 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:25.677 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:41.724 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:25.691 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:41.735 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:25.701 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:41.739 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:25.678 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:41.797 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:25.763 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:41.809 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:25.748 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:41.853 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:25.792 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:41.856 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:25.822 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:41.872 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:25.838 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:51.884 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:35.850 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:51.951 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:35.917 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:51.994 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:35.960 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:52.010 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:35.976 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:52.012 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:35.951 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:52.016 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:35.956 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:52.019 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:35.985 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:52.051 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:35.990 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:52.058 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:36.024 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:56.734 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:40.700 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:56.779 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:40.745 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:56.783 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:40.722 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:56.786 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:40.725 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:56.799 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:40.765 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:56.802 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:40.741 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:56.831 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:40.797 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:56.833 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:40.799 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:44:56.846 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:40.812 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:11.783 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:55.749 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:11.831 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:55.797 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:11.851 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:13:55.817 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:11.879 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:55.818 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:11.881 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:55.820 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:11.891 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:55.857 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:11.896 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:55.862 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:11.897 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:55.837 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:11.915 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:13:55.881 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:16.926 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:00.892 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:16.931 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:00.897 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:16.946 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:00.912 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:16.993 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:00.932 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:16.996 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:00.962 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:16.996 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:00.935 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:16.997 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:00.963 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:17.009 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:00.975 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:17.010 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:00.949 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:32.076 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:16.042 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:32.080 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:16.046 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:32.080 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:16.019 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:32.085 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:16.024 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:32.109 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:16.075 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:32.110 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:16.049 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:32.148 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:16.113 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:32.152 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:16.118 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:32.179 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:16.145 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:52.065 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:36.030 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:52.073 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:36.039 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:52.112 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:36.077 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:52.128 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:36.067 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:52.133 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:36.072 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:52.137 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:36.103 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:52.141 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:36.107 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:52.162 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:36.101 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:52.212 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:36.178 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:57.012 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:40.978 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:57.064 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:41.030 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:57.065 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:41.004 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:57.067 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:41.007 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:57.081 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:41.046 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:57.081 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:41.020 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:57.118 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:41.084 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:57.119 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:41.085 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:45:57.134 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:41.099 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:12.024 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:55.990 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:12.028 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:55.994 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:12.052 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:14:56.018 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:12.072 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:56.011 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:12.076 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:56.015 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:12.093 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:56.059 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:12.095 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:56.061 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:12.095 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:56.034 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:12.118 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:14:56.084 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:17.041 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:01.007 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:17.046 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:01.012 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:17.057 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:01.023 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:17.092 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:01.058 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:17.095 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:01.061 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:17.097 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:01.036 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:17.101 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:01.041 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:17.106 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:01.072 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:17.114 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:01.053 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:32.238 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:16.204 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:32.241 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:16.207 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:32.264 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:16.230 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:32.287 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:16.252 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:32.290 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:16.257 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:32.291 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:16.231 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:32.296 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:16.235 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:32.311 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:16.277 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:32.315 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:16.254 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:37.061 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:21.027 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:37.107 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:21.073 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:37.124 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:21.089 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:37.161 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:21.100 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:37.162 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:21.101 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:37.169 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:21.135 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:37.172 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:21.138 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:37.172 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:21.111 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:37.184 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:21.150 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.265 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:31.231 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.273 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:31.239 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.303 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:31.269 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.334 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:31.273 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.335 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:31.274 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.340 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:31.306 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.341 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:31.307 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.360 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:31.299 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.367 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:31.333 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.826 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:31.765 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.827 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:31.766 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.835 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:31.775 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.892 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:31.858 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.893 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:31.859 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.898 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:31.865 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.899 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:31.866 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.901 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:31.867 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:47.908 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:31.874 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:57.257 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:41.222 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:57.304 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:41.270 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:57.307 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:41.246 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:57.308 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:41.248 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:57.329 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:41.294 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:57.330 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:41.269 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:57.350 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:41.316 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:57.354 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:41.320 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:57.375 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:41.341 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:58.941 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:42.907 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:58.945 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:42.911 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:58.947 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:42.887 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:58.949 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:42.888 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:58.951 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:42.917 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:58.953 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:42.919 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:58.962 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:42.928 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:58.964 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:42.903 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:46:58.966 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:42.933 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:02.298 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:46.260 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:02.299 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:46.264 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:02.309 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:15:46.274 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:02.344 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:46.310 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:02.346 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:46.285 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:02.347 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:46.286 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:02.347 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:46.313 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:02.356 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:46.295 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:02.358 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:15:46.324 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:17.333 INFO:tasks.ceph.osd.2.smithi170.stderr:2023-10-30T17:47:17.307+0000 22e89640 -1 snap_mapper.add_oid found existing snaps mapped on 219:405e84fd:test-rados-api-smithi170-91745-80::foo:2, removing 2023-10-30T17:47:17.341 INFO:tasks.ceph.osd.5.smithi177.stderr:2023-10-30T17:47:17.334+0000 26e91640 -1 snap_mapper.add_oid found existing snaps mapped on 219:405e84fd:test-rados-api-smithi170-91745-80::foo:2, removing 2023-10-30T17:47:17.363 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:01.329 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:17.375 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:01.341 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:17.404 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:01.370 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:17.416 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:01.356 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:17.418 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:01.358 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:17.423 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:01.389 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:17.426 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:01.392 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:17.436 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:01.375 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:17.443 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:01.409 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:22.393 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:06.359 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:22.395 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:06.361 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:22.417 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:06.383 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:22.437 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:06.376 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:22.439 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:06.379 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:22.448 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:06.414 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:22.451 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:06.417 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:22.456 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:06.396 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:22.465 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:06.431 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:37.440 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:21.406 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:37.442 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:21.408 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:37.472 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:21.438 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:37.489 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:21.455 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:37.491 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:21.430 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:37.494 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:21.433 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:37.495 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:21.461 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:37.522 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:21.462 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:37.525 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:21.491 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:38.211 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:22.150 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:38.212 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:22.152 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:38.221 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:22.161 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:38.281 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:22.247 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:38.283 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:22.249 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:38.283 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:22.249 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:38.285 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:22.251 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:38.296 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:22.262 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:38.298 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:22.264 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:52.584 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:36.549 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:52.587 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:36.553 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:52.614 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:36.580 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:52.630 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:36.570 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:52.634 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:36.573 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:52.635 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:36.601 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:52.638 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:36.604 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:52.659 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:36.598 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:52.676 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:36.642 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:53.209 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (20584 ms) 2023-10-30T17:47:53.209 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-10-30T17:47:53.210 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25681 ms) 2023-10-30T17:47:53.210 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-10-30T17:47:53.210 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3947 ms) 2023-10-30T17:47:53.210 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-10-30T17:47:53.211 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.211 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5186 ms) 2023-10-30T17:47:53.211 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-10-30T17:47:53.211 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4412 ms) 2023-10-30T17:47:53.211 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-10-30T17:47:53.211 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.212 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (5122 ms) 2023-10-30T17:47:53.212 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-10-30T17:47:53.212 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.212 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5323 ms) 2023-10-30T17:47:53.212 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-10-30T17:47:53.213 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.213 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40329 ms) 2023-10-30T17:47:53.213 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-10-30T17:47:53.213 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.213 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17775 ms) 2023-10-30T17:47:53.214 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-10-30T17:47:53.214 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.214 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5577 ms) 2023-10-30T17:47:53.214 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-10-30T17:47:53.214 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.215 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4461 ms) 2023-10-30T17:47:53.215 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-10-30T17:47:53.215 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.215 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26118 ms) 2023-10-30T17:47:53.215 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-10-30T17:47:53.216 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.216 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16272 ms) 2023-10-30T17:47:53.216 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-10-30T17:47:53.216 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.216 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6642 ms) 2023-10-30T17:47:53.216 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-10-30T17:47:53.217 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.217 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5213 ms) 2023-10-30T17:47:53.217 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-10-30T17:47:53.217 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.217 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: waiting for scrubs... 2023-10-30T17:47:53.218 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: done waiting 2023-10-30T17:47:53.218 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26252 ms) 2023-10-30T17:47:53.218 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-10-30T17:47:53.218 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.218 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10683 ms) 2023-10-30T17:47:53.219 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-10-30T17:47:53.219 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.219 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (10926 ms) 2023-10-30T17:47:53.219 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-10-30T17:47:53.219 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:47:53.220 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (10714 ms) 2023-10-30T17:47:53.220 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-10-30T17:47:57.708 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:41.674 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:57.710 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:41.677 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:57.734 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:41.700 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:57.766 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:41.732 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:57.768 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:41.707 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:57.768 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:41.734 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:57.777 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:41.716 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:57.783 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:41.749 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:47:57.799 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:41.738 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:07.614 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:51.580 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:07.619 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:51.585 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:07.641 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:16:51.607 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:07.667 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:51.633 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:07.672 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:51.638 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:07.677 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:51.616 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:07.678 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:51.618 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:07.693 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:16:51.659 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:07.699 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:51.638 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:17.785 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:01.751 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:17.788 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:01.754 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:17.791 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:01.757 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:17.821 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:01.787 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:17.865 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:01.831 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:17.891 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:01.857 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:17.916 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:01.855 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:17.921 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:01.861 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:17.948 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:01.888 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:22.673 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:06.639 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:22.677 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:06.643 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:22.691 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:06.657 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:22.726 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:06.666 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:22.727 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:06.693 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:22.729 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:06.695 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:22.741 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:06.707 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:22.745 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:06.684 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:22.758 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:06.698 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:37.860 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:21.826 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:37.862 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:21.828 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:37.878 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:21.844 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:37.907 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:21.846 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:37.908 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:21.847 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:37.911 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:21.877 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:37.912 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:21.879 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:37.924 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:21.864 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:37.930 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:21.896 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:42.834 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:26.800 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:42.839 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:26.805 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:42.855 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:26.821 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:42.895 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:26.835 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:42.898 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:26.864 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:42.902 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:26.841 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:42.903 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:26.869 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:42.916 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:26.881 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:42.918 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:26.857 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:48:56.742 INFO:tasks.ceph.osd.1.smithi170.stderr:2023-10-30T17:48:56.732+0000 22688640 -1 snap_mapper.add_oid found existing snaps mapped on 83:3cb0aa8c:test-rados-api-smithi170-91745-97::foo:31, removing 2023-10-30T17:48:56.774 INFO:tasks.ceph.osd.6.smithi177.stderr:2023-10-30T17:48:56.762+0000 2be9b640 -1 snap_mapper.add_oid found existing snaps mapped on 83:3cb0aa8c:test-rados-api-smithi170-91745-97::foo:31, removing 2023-10-30T17:49:02.909 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:46.875 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:02.913 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:46.879 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:02.940 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:46.906 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:02.960 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:46.900 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:02.963 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:46.930 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:02.966 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:46.932 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:02.982 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:46.921 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:02.995 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:46.961 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:03.006 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:46.946 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:11.084 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:55.050 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:11.086 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:55.052 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:11.108 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:17:55.074 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:11.131 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:55.070 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:11.133 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:55.073 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:11.143 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:55.109 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:11.144 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:55.110 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:11.155 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:55.095 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:11.168 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:17:55.134 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:17.846 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:01.811 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:17.847 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:01.813 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:17.877 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:01.843 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:17.899 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:01.838 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:17.902 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:01.842 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:17.909 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:01.875 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:17.913 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:01.879 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:17.926 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:01.866 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:17.931 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:01.897 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:22.950 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:06.916 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:22.953 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:06.919 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:22.973 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:06.939 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:23.010 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:06.949 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:23.013 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:06.953 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:23.019 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:06.985 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:23.022 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:06.988 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:23.032 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:06.971 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:23.045 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:07.011 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.010 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:16.976 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.012 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:16.978 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.052 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:17.018 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.068 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:17.007 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.072 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:17.011 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.088 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:17.054 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.090 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:17.056 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.109 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:17.048 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.154 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:17.120 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.691 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:17.630 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.691 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:17.631 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.699 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:17.638 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.772 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:17.738 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.773 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:17.739 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.782 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:17.748 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.832 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:17.798 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.883 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:17.849 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:33.894 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:17.860 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:42.989 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:26.955 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:42.993 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:26.959 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:43.017 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:26.983 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:43.039 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:27.005 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:43.040 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:26.980 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:43.042 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:26.981 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:43.044 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:27.010 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:43.062 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:27.002 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:43.067 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:27.033 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:53.130 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:37.096 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:53.182 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:37.148 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:53.210 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:37.177 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:53.247 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:37.186 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:53.249 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:37.215 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:53.250 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:37.190 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:53.270 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:37.236 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:53.286 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:37.225 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:49:53.314 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:37.280 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:03.172 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:47.138 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:03.176 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:47.142 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:03.209 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:47.175 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:03.222 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:47.162 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:03.226 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:47.165 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:03.241 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:47.208 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:03.243 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:47.209 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:03.249 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:47.188 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:03.269 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:47.235 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:08.128 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:52.094 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:08.180 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:52.146 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:08.196 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:18:52.162 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:08.227 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:52.166 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:08.230 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:52.196 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:08.231 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:52.171 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:08.232 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:52.198 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:08.245 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:52.184 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:08.246 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:18:52.212 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.153 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:07.119 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.155 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:07.121 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.157 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:07.123 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.199 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:07.165 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.203 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:07.143 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.208 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:07.147 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.220 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:07.186 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.240 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:07.180 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.251 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:07.217 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.875 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:07.815 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.876 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:07.816 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.884 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:07.823 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.928 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:07.894 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.929 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:07.895 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.940 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:07.906 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.948 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:07.914 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.949 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:07.915 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:23.960 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:07.927 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:38.291 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:22.257 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:38.296 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:22.262 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:38.325 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:22.291 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:38.340 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:22.306 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:38.342 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:22.309 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:38.343 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:22.282 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:38.344 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:22.284 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:38.372 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:22.311 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:38.373 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:22.339 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:43.358 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:27.324 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:43.361 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:27.327 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:43.373 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:27.339 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:43.408 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:27.347 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:43.410 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:27.349 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:43.412 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:27.378 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:43.416 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:27.382 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:43.422 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:27.362 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:43.429 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:27.395 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:53.237 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:37.203 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:53.284 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:37.250 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:53.285 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:37.223 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:53.290 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:37.230 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:53.304 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:37.243 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:53.325 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:37.291 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:53.330 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:37.297 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:53.335 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:37.301 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:50:53.351 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:37.317 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:03.296 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:47.261 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:03.297 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:47.263 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:03.298 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:47.237 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:03.312 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:47.278 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:03.322 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:47.261 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:03.337 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:47.277 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:03.343 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:47.309 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:03.349 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:47.315 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:03.414 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:47.380 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:13.440 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:57.405 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:13.441 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:57.407 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:13.470 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:57.436 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:13.488 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:57.427 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:13.489 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:57.429 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:13.505 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:57.471 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:13.506 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:57.472 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:13.507 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:57.446 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:13.531 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:57.497 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:14.187 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:58.126 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:14.188 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:58.127 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:14.199 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:58.139 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:14.314 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:58.280 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:14.314 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:58.280 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:14.315 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:58.281 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:14.317 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:58.283 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:14.331 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:19:58.297 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:14.421 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:19:58.387 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:23.492 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:07.458 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:23.496 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:07.462 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:23.530 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:07.496 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:23.538 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:07.477 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:23.539 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:07.478 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:23.551 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:07.517 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:23.557 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:07.523 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:23.562 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:07.501 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:23.581 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:07.547 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:33.448 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:17.414 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:33.495 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:17.435 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:33.498 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:17.464 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:33.502 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:17.442 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:33.523 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:17.462 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:33.533 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:17.499 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:33.553 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:17.519 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:33.554 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:17.520 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:33.579 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:17.545 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:43.572 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:27.538 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:43.577 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:27.543 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:43.599 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:27.565 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:43.624 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:27.563 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:43.625 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:27.564 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:43.628 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:27.594 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:43.629 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:27.595 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:43.647 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:27.587 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:43.656 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:27.622 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:48.473 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:32.439 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:48.520 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:32.459 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:48.523 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:32.462 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:48.523 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:32.489 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:48.550 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:32.489 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:48.568 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:32.534 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:48.570 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:32.536 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:48.580 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:32.546 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:48.612 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:32.578 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:51:59.637 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:51:59.638 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (10517 ms) 2023-10-30T17:51:59.638 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-10-30T17:51:59.638 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:51:59.638 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8162 ms) 2023-10-30T17:51:59.639 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-10-30T17:51:59.639 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:51:59.639 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6527 ms) 2023-10-30T17:51:59.639 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-10-30T17:51:59.639 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:51:59.639 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26726 ms) 2023-10-30T17:51:59.640 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-10-30T17:51:59.640 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:51:59.640 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15287 ms) 2023-10-30T17:51:59.640 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-10-30T17:51:59.640 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (13248 ms) 2023-10-30T17:51:59.641 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-10-30T17:51:59.641 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-10-30T17:51:59.641 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: 00000010 69 73 |is| 2023-10-30T17:51:59.641 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: 00000012 2023-10-30T17:51:59.641 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12663 ms) 2023-10-30T17:51:59.642 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-10-30T17:51:59.642 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T17:51:59.642 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7668 ms) 2023-10-30T17:51:59.642 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (811336 ms total) 2023-10-30T17:51:59.642 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: 2023-10-30T17:51:59.642 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-10-30T17:51:59.643 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-10-30T17:51:59.643 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1641 ms) 2023-10-30T17:51:59.643 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-10-30T17:51:59.643 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (12094 ms) 2023-10-30T17:51:59.644 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-10-30T17:51:59.644 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18911 ms) 2023-10-30T17:51:59.644 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-10-30T17:51:59.644 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (16 ms) 2023-10-30T17:51:59.644 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (32662 ms total) 2023-10-30T17:51:59.644 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: 2023-10-30T17:51:59.645 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-10-30T17:51:59.645 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-10-30T17:51:59.645 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8705 ms) 2023-10-30T17:51:59.645 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-10-30T17:51:59.646 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8953 ms) 2023-10-30T17:51:59.646 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-10-30T17:51:59.646 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: waiting for scrub... 2023-10-30T17:51:59.646 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: done waiting 2023-10-30T17:51:59.647 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (35865 ms) 2023-10-30T17:51:59.647 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-10-30T17:51:59.647 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (11141 ms) 2023-10-30T17:51:59.647 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-10-30T17:51:59.647 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9207 ms) 2023-10-30T17:51:59.647 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-10-30T17:51:59.648 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8788 ms) 2023-10-30T17:51:59.648 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-10-30T17:51:59.648 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11689 ms) 2023-10-30T17:51:59.648 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-10-30T17:52:03.532 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:47.498 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:03.535 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:47.501 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:03.568 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:47.534 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:03.582 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:47.521 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:03.583 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:47.523 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:03.587 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:47.553 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:03.589 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:47.554 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:03.612 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:47.552 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:03.620 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:47.586 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:04.113 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:48.052 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:04.114 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:48.053 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:04.122 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:48.061 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:04.281 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:48.247 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:04.282 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:48.248 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:04.293 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:20:48.259 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:04.351 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:48.317 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:04.352 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:48.318 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:04.361 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:20:48.327 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:18.612 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:02.578 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:18.614 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:02.581 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:18.641 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:02.607 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:18.657 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:02.597 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:18.659 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:02.599 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:18.672 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:02.639 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:18.677 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:02.643 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:18.689 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:02.628 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:18.713 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:02.679 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:23.694 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:07.660 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:23.696 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:07.662 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:23.710 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:07.676 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:23.741 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:07.680 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:23.744 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:07.710 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:23.744 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:07.684 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:23.760 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:07.699 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:23.763 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:07.729 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:23.786 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:07.752 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:33.647 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:17.613 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:33.649 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:17.588 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:33.649 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:17.615 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:33.653 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:17.592 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:33.677 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:17.616 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:33.679 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:17.645 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:33.698 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:17.664 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:33.705 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:17.671 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:33.732 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:17.698 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:43.806 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:27.772 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:43.810 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:27.776 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:43.834 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:27.800 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:43.855 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:27.794 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:43.856 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:27.795 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:43.875 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:27.841 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:43.880 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:27.819 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:43.893 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:27.859 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:43.920 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:27.886 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:48.789 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:32.755 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:48.791 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:32.757 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:48.806 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:32.772 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:48.857 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:32.823 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:48.858 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:32.797 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:48.859 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:32.798 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:48.864 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:32.830 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:48.878 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:32.817 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:52:48.886 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:32.852 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:03.705 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:47.671 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:03.751 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:47.691 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:03.754 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:47.694 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:03.757 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:47.723 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:03.783 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:47.722 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:03.792 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:47.758 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:03.808 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:47.774 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:03.811 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:47.777 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:03.841 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:47.807 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:04.411 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:48.350 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:04.412 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:48.351 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:04.419 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:48.358 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:04.631 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:48.596 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:04.634 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:48.600 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:04.646 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:48.612 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:04.694 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:48.660 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:04.695 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:48.661 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:04.704 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:48.670 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:10.321 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:54.260 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:10.328 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:54.294 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:10.330 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:54.296 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:10.360 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:54.326 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:10.366 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:54.332 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:10.372 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:54.311 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:10.389 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:21:54.355 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:10.398 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:54.337 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:10.399 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:21:54.365 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:18.817 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:02.783 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:18.819 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:02.785 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:18.847 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:02.813 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:18.859 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:02.798 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:18.864 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:02.804 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:18.875 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:02.841 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:18.877 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:02.844 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:18.884 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:02.823 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:18.906 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:02.872 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:23.782 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:07.748 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:23.831 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:07.797 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:23.843 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:07.809 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:23.874 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:07.814 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:23.876 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:07.816 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:23.877 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:07.843 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:23.880 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:07.846 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:23.887 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:07.826 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:23.891 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:07.857 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:43.930 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:27.896 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:43.931 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:27.871 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:43.935 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:27.901 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:43.935 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:27.874 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:43.962 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:27.901 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:43.982 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:27.948 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:43.997 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:27.963 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:44.000 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:27.966 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:44.029 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:27.995 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:48.946 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:32.912 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:48.951 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:32.917 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:48.974 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:32.940 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:48.996 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:32.935 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:48.999 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:32.938 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:49.013 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:32.953 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:49.020 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:32.987 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:49.024 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:32.990 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:49.042 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:33.007 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:59.080 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:43.046 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:59.083 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:43.049 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:59.108 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:43.074 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:59.126 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:43.065 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:59.127 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:43.066 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:59.136 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:43.101 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:59.137 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:43.103 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:59.150 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:43.089 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:53:59.158 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:43.124 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:09.017 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:52.983 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:09.019 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:52.985 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:09.034 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:53.000 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:09.065 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:53.031 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:09.065 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:53.005 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:09.067 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:53.007 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:09.068 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:53.034 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:09.079 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:53.018 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:09.081 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:53.047 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:13.994 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:57.960 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:14.048 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:57.987 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:14.050 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:57.990 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:14.065 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:58.004 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:14.067 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:58.033 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:14.098 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:22:58.064 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:14.128 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:58.094 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:14.133 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:58.097 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:14.147 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:22:58.113 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:29.219 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:13.185 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:29.223 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:13.189 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:29.250 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:13.216 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:29.266 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:13.205 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:29.269 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:13.208 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:29.277 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:13.243 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:29.278 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:13.244 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:29.292 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:13.231 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:29.304 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:13.270 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:44.083 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:28.049 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:44.129 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:28.068 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:44.130 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:28.069 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:44.131 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:28.097 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:44.157 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:28.096 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:44.160 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:28.126 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:44.198 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:28.164 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:44.200 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:28.166 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:44.226 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:28.192 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:49.293 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:33.259 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:49.351 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:33.290 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:49.352 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:33.291 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:49.360 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:33.326 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:49.363 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:33.329 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:49.372 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:33.338 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:49.376 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:33.315 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:49.385 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:33.351 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:49.400 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:33.366 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:59.353 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:43.319 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:59.355 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:43.322 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:59.390 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:43.356 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:59.400 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:43.339 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:59.401 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:43.341 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:59.415 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:43.380 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:59.419 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:43.385 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:59.424 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:43.363 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:54:59.444 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:43.410 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:00.356 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (9469 ms) 2023-10-30T17:55:00.357 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-10-30T17:55:00.357 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (15449 ms) 2023-10-30T17:55:00.357 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-10-30T17:55:00.357 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (10282 ms) 2023-10-30T17:55:00.357 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-10-30T17:55:00.358 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15065 ms) 2023-10-30T17:55:00.358 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-10-30T17:55:00.358 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9794 ms) 2023-10-30T17:55:00.358 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-10-30T17:55:00.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9894 ms) 2023-10-30T17:55:00.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-10-30T17:55:00.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: hmm, no HitSet yet 2023-10-30T17:55:00.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2023-10-30T17:55:00.359 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (10269 ms) 2023-10-30T17:55:00.360 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-10-30T17:55:00.360 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,0 2023-10-30T17:55:00.360 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: first is 1698688399 2023-10-30T17:55:00.360 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,0 2023-10-30T17:55:00.360 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,0 2023-10-30T17:55:00.361 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,1698688401,1698688402,0 2023-10-30T17:55:00.361 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,1698688401,1698688402,0 2023-10-30T17:55:00.361 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,1698688401,1698688402,0 2023-10-30T17:55:00.361 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,1698688401,1698688402,1698688404,1698688405,0 2023-10-30T17:55:00.361 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,1698688401,1698688402,1698688404,1698688405,0 2023-10-30T17:55:00.362 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,1698688401,1698688402,1698688404,1698688405,0 2023-10-30T17:55:00.362 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,1698688401,1698688402,1698688404,1698688405,1698688407,1698688408,0 2023-10-30T17:55:00.362 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,1698688401,1698688402,1698688404,1698688405,1698688407,1698688408,0 2023-10-30T17:55:00.362 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688399,1698688401,1698688402,1698688404,1698688405,1698688407,1698688408,0 2023-10-30T17:55:00.362 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: got ls 1698688402,1698688404,1698688405,1698688407,1698688408,1698688410,1698688411,0 2023-10-30T17:55:00.363 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: first now 1698688402, trimmed 2023-10-30T17:55:00.363 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22910 ms) 2023-10-30T17:55:00.363 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-10-30T17:55:00.363 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: foo0 2023-10-30T17:55:00.363 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-10-30T17:55:00.364 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (18267 ms) 2023-10-30T17:55:00.364 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-10-30T17:55:00.364 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (21136 ms) 2023-10-30T17:55:00.364 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-10-30T17:55:00.364 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (25002 ms) 2023-10-30T17:55:00.365 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-10-30T17:55:00.365 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5470 ms) 2023-10-30T17:55:00.365 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-10-30T17:55:00.365 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4768 ms) 2023-10-30T17:55:00.365 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-10-30T17:55:00.366 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4512 ms) 2023-10-30T17:55:00.366 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-10-30T17:55:00.366 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4310 ms) 2023-10-30T17:55:00.366 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (280945 ms total) 2023-10-30T17:55:00.367 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: 2023-10-30T17:55:00.367 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-10-30T17:55:00.367 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1286536 ms total) 2023-10-30T17:55:00.367 INFO:tasks.workunit.client.0.smithi170.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-10-30T17:55:00.373 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.373 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91470 2023-10-30T17:55:00.374 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91470 2023-10-30T17:55:00.374 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.374 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91654 2023-10-30T17:55:00.374 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91654 2023-10-30T17:55:00.374 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.375 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92193 2023-10-30T17:55:00.375 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 92193 2023-10-30T17:55:00.375 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.375 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91911 2023-10-30T17:55:00.375 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91911 2023-10-30T17:55:00.376 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.376 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91817 2023-10-30T17:55:00.376 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91817 2023-10-30T17:55:00.376 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.376 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91549 2023-10-30T17:55:00.376 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91549 2023-10-30T17:55:00.377 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.377 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91854 2023-10-30T17:55:00.377 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91854 2023-10-30T17:55:00.377 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.377 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91572 2023-10-30T17:55:00.378 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91572 2023-10-30T17:55:00.378 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.378 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92084 2023-10-30T17:55:00.378 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 92084 2023-10-30T17:55:00.378 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.379 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91444 2023-10-30T17:55:00.379 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91444 2023-10-30T17:55:00.379 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.379 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91504 2023-10-30T17:55:00.379 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91504 2023-10-30T17:55:00.380 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.380 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92101 2023-10-30T17:55:00.380 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 92101 2023-10-30T17:55:00.380 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.380 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91435 2023-10-30T17:55:00.380 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91435 2023-10-30T17:55:00.381 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.381 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91456 2023-10-30T17:55:00.381 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91456 2023-10-30T17:55:00.381 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.382 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91688 2023-10-30T17:55:00.382 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91688 2023-10-30T17:55:00.382 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.382 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91830 2023-10-30T17:55:00.382 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91830 2023-10-30T17:55:00.383 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.383 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91753 2023-10-30T17:55:00.383 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91753 2023-10-30T17:55:00.383 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.383 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92177 2023-10-30T17:55:00.383 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 92177 2023-10-30T17:55:00.384 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.384 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92055 2023-10-30T17:55:00.384 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 92055 2023-10-30T17:55:00.384 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.384 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91607 2023-10-30T17:55:00.385 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91607 2023-10-30T17:55:00.385 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.385 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91944 2023-10-30T17:55:00.385 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91944 2023-10-30T17:55:00.385 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.385 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91428 2023-10-30T17:55:00.386 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91428 2023-10-30T17:55:00.386 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.386 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91487 2023-10-30T17:55:00.386 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91487 2023-10-30T17:55:00.386 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.387 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91870 2023-10-30T17:55:00.387 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91870 2023-10-30T17:55:00.387 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.387 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=91527 2023-10-30T17:55:00.387 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 91527 2023-10-30T17:55:00.388 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.388 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92000 2023-10-30T17:55:00.388 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 92000 2023-10-30T17:55:00.388 INFO:tasks.workunit.client.0.smithi170.stderr:+ for t in "${!pids[@]}" 2023-10-30T17:55:00.388 INFO:tasks.workunit.client.0.smithi170.stderr:+ pid=92021 2023-10-30T17:55:00.388 INFO:tasks.workunit.client.0.smithi170.stderr:+ wait 92021 2023-10-30T17:55:00.389 INFO:tasks.workunit.client.0.smithi170.stderr:+ exit 0 2023-10-30T17:55:00.389 INFO:tasks.workunit.client.0.smithi170.stderr:+ cleanup 2023-10-30T17:55:00.389 INFO:tasks.workunit.client.0.smithi170.stderr:+ pkill -P 91423 2023-10-30T17:55:00.390 INFO:tasks.workunit.client.0.smithi170.stderr:+ true 2023-10-30T17:55:00.390 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T17:55:00.391 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T17:55:00.419 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-10-30T17:55:00.420 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T17:55:00.732 DEBUG:teuthology.parallel:result is None 2023-10-30T17:55:00.733 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T17:55:00.759 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T17:55:00.760 DEBUG:teuthology.orchestra.run.smithi170:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-10-30T17:55:00.824 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T17:55:00.825 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T17:55:00.834 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T17:55:00.834 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T17:55:00.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.962+0000 7f9997a51640 10 monclient: get_monmap_and_config 2023-10-30T17:55:00.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.962+0000 7f9997a51640 10 monclient: build_initial_monmap 2023-10-30T17:55:00.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.963+0000 7f9997a51640 10 monclient: monmap: 2023-10-30T17:55:00.965 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:00.966 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:00.966 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:00.963922+0000 2023-10-30T17:55:00.966 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:00.963922+0000 2023-10-30T17:55:00.967 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:00.967 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:00.967 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:00.968 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:00.968 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:00.968 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:00.969 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:00.969 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:00.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.963+0000 7f9997a51640 1 Processor -- start 2023-10-30T17:55:00.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.963+0000 7f9997a51640 1 -- start start 2023-10-30T17:55:00.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.963+0000 7f9997a51640 10 monclient: init 2023-10-30T17:55:00.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:00.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-10-30T17:55:00.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f99900b0330 0x7f99900a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:00.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 10 monclient(hunting): picked mon.noname-c con 0x7f99900b0330 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:00.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a8c00 0x7f99900a6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:00.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 10 monclient(hunting): picked mon.noname-b con 0x7f99900a8c00 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:00.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99900a7050 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:00.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900a82b0 con 0x7f99900a7050 2023-10-30T17:55:00.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:00.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99900b0710 con 0x7f99900b0330 2023-10-30T17:55:00.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:00.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99900a74e0 con 0x7f99900a8c00 2023-10-30T17:55:00.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:00.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.964+0000 7f9997a51640 10 monclient(hunting): authenticate will time out at 2683.287541s 2023-10-30T17:55:00.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f9994fec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a8c00 0x7f99900a6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:00.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f9994fec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a8c00 0x7f99900a6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:38836/0 (socket says 172.21.15.170:38836) 2023-10-30T17:55:00.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f9994fec640 1 -- 172.21.15.170:0/1307445479 learned_addr learned my addr 172.21.15.170:0/1307445479 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:00.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f99957ed640 1 --2- 172.21.15.170:0/1307445479 >> v2:172.21.15.177:3300/0 conn(0x7f99900b0330 0x7f99900a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:00.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f9994fec640 10 monclient(hunting): get_auth_request con 0x7f99900a8c00 auth_method 0 2023-10-30T17:55:00.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f9994fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:00.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f9994fec640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:00.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f9995fee640 1 --1- 172.21.15.170:0/1307445479 >> v1:172.21.15.177:6789/0 conn(0x7f99900a7050 0x7f999013cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:45880/0 (socket says 172.21.15.170:45880) 2023-10-30T17:55:00.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f9994fec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:00.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f99957ed640 10 monclient(hunting): get_auth_request con 0x7f99900b0330 auth_method 0 2023-10-30T17:55:00.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f99957ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:00.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f99957ed640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:00.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.965+0000 7f99957ed640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:00.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.966+0000 7f9994fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:00.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.966+0000 7f9994fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:00.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.966+0000 7f9994fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:00.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.966+0000 7f99957ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:00.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.966+0000 7f99957ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:00.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.966+0000 7f99957ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:00.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.967+0000 7f9994fec640 10 monclient(hunting): handle_auth_done global_id 16289 payload 274 2023-10-30T17:55:00.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.967+0000 7f9994fec640 1 -- 172.21.15.170:0/1307445479 >> v2:172.21.15.177:3300/0 conn(0x7f99900b0330 msgr2=0x7f99900a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:00.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.967+0000 7f9994fec640 1 --2- 172.21.15.170:0/1307445479 >> v2:172.21.15.177:3300/0 conn(0x7f99900b0330 0x7f99900a5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:00.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.967+0000 7f9994fec640 1 -- 172.21.15.170:0/1307445479 >> v1:172.21.15.177:6789/0 conn(0x7f99900a7050 legacy=0x7f999013cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:00.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.967+0000 7f9994fec640 10 monclient: _finish_hunting 0 2023-10-30T17:55:00.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.967+0000 7f9994fec640 1 monclient: found mon.noname-b 2023-10-30T17:55:00.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.967+0000 7f9994fec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:00.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.967+0000 7f9994fec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:55:00.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.967+0000 7f9994fec640 1 -- 172.21.15.170:0/1307445479 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99900aac60 con 0x7f99900a8c00 2023-10-30T17:55:00.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.969+0000 7f9994fec640 1 --2- 172.21.15.170:0/1307445479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a8c00 0x7f99900a6b10 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f9980009870 tx=0x7f998002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6726238ad537d795 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:01.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 1 -- 172.21.15.170:0/1307445479 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9980031640 con 0x7f99900a8c00 2023-10-30T17:55:01.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:01.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:01.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 10 monclient: dump: 2023-10-30T17:55:01.002 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:01.002 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:01.002 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.003 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.003 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:01.004 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.004 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:01.004 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:01.005 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:01.005 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 10 monclient: _finish_auth 0 2023-10-30T17:55:01.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 10 monclient: _check_auth_tickets 2023-10-30T17:55:01.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:01.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 1 -- 172.21.15.170:0/1307445479 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99800317a0 con 0x7f99900a8c00 2023-10-30T17:55:01.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9997a51640 5 monclient: authenticate success, global_id 16289 2023-10-30T17:55:01.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9997a51640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:01.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:01.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9997a51640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:01.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 1 -- 172.21.15.170:0/1307445479 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9980031b30 con 0x7f99900a8c00 2023-10-30T17:55:01.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:01.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9997a51640 4 set_mon_vals no callback set 2023-10-30T17:55:01.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:55:01.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.972+0000 7f9987fff640 10 monclient: dump: 2023-10-30T17:55:01.010 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:01.011 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:01.011 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.012 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.012 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:01.012 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.013 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:01.013 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:01.013 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:01.014 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.973+0000 7f9997a51640 10 monclient: shutdown 2023-10-30T17:55:01.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.973+0000 7f9997a51640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:01.014 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.973+0000 7f9997a51640 1 -- 172.21.15.170:0/1307445479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a8c00 msgr2=0x7f99900a6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:01.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.973+0000 7f9997a51640 1 --2- 172.21.15.170:0/1307445479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a8c00 0x7f99900a6b10 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f9980009870 tx=0x7f998002b8f0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.973+0000 7f9997a51640 1 -- 172.21.15.170:0/1307445479 shutdown_connections 2023-10-30T17:55:01.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.973+0000 7f9997a51640 1 --2- 172.21.15.170:0/1307445479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a8c00 0x7f99900a6b10 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.973+0000 7f9997a51640 1 --2- 172.21.15.170:0/1307445479 >> v2:172.21.15.177:3300/0 conn(0x7f99900b0330 0x7f99900a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.015 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.973+0000 7f9997a51640 1 -- 172.21.15.170:0/1307445479 >> 172.21.15.170:0/1307445479 conn(0x7f99900b3560 msgr2=0x7f99900af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:01.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.973+0000 7f9997a51640 1 -- 172.21.15.170:0/1307445479 shutdown_connections 2023-10-30T17:55:01.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.973+0000 7f9997a51640 1 -- 172.21.15.170:0/1307445479 wait complete. 2023-10-30T17:55:01.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.974+0000 7f9997a51640 10 monclient: build_initial_monmap 2023-10-30T17:55:01.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.974+0000 7f9997a51640 10 monclient: monmap: 2023-10-30T17:55:01.016 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:01.017 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:01.017 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:01.017 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:01.017 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:01.017 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.018 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:01.018 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:01.018 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:01.018 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.974+0000 7f9997a51640 1 librados: starting msgr at 2023-10-30T17:55:01.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.974+0000 7f9997a51640 1 librados: starting objecter 2023-10-30T17:55:01.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.974+0000 7f9997a51640 1 Processor -- start 2023-10-30T17:55:01.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.974+0000 7f9997a51640 1 -- start start 2023-10-30T17:55:01.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.974+0000 7f9997a51640 1 librados: setting wanted keys 2023-10-30T17:55:01.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.974+0000 7f9997a51640 1 librados: calling monclient init 2023-10-30T17:55:01.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.974+0000 7f9997a51640 10 monclient: init 2023-10-30T17:55:01.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:01.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:55:01.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a7050 0x7f9990070b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:01.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 10 monclient(hunting): picked mon.noname-c con 0x7f99900a7050 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:01.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900a8c00 0x7f99901d4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:01.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 10 monclient(hunting): picked mon.noname-b con 0x7f99900a8c00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:01.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99900b0330 0x7f9990138c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:01.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 10 monclient(hunting): picked mon.noname-a con 0x7f99900b0330 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:01.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:01.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99901d4210 con 0x7f99900b0330 2023-10-30T17:55:01.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:01.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99901d4460 con 0x7f99900a8c00 2023-10-30T17:55:01.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:01.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999006d410 con 0x7f99900a7050 2023-10-30T17:55:01.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:01.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9997a51640 10 monclient(hunting): authenticate will time out at 2683.298531s 2023-10-30T17:55:01.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f99957ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a7050 0x7f9990070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:01.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.975+0000 7f9995fee640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99900b0330 0x7f9990138c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:01.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9994fec640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900a8c00 0x7f99901d4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:01.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f99957ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a7050 0x7f9990070b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:38842/0 (socket says 172.21.15.170:38842) 2023-10-30T17:55:01.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f99957ed640 1 -- 172.21.15.170:0/2315319637 learned_addr learned my addr 172.21.15.170:0/2315319637 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:01.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f99957ed640 10 monclient(hunting): get_auth_request con 0x7f99900a7050 auth_method 0 2023-10-30T17:55:01.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f99957ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:01.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f99957ed640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:01.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f99957ed640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:01.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9995fee640 10 monclient(hunting): get_auth_request con 0x7f99900b0330 auth_method 0 2023-10-30T17:55:01.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9995fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:01.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9995fee640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:01.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9995fee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:01.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9994fec640 10 monclient(hunting): get_auth_request con 0x7f99900a8c00 auth_method 0 2023-10-30T17:55:01.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9994fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:01.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9994fec640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:01.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9994fec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:01.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f99957ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:01.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f99957ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:01.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f99957ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:01.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9994fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:01.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9994fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:01.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9994fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:01.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9995fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:01.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9995fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:01.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.976+0000 7f9995fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:01.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.979+0000 7f9994fec640 10 monclient(hunting): handle_auth_done global_id 21424 payload 824 2023-10-30T17:55:01.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.979+0000 7f9994fec640 1 -- 172.21.15.170:0/2315319637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a7050 msgr2=0x7f9990070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:01.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.979+0000 7f9994fec640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a7050 0x7f9990070b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.979+0000 7f9994fec640 1 -- 172.21.15.170:0/2315319637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99900b0330 msgr2=0x7f9990138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:01.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.979+0000 7f9994fec640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99900b0330 0x7f9990138c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.979+0000 7f9994fec640 10 monclient: _finish_hunting 0 2023-10-30T17:55:01.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.979+0000 7f9994fec640 1 monclient: found mon.noname-b 2023-10-30T17:55:01.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.979+0000 7f9994fec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:01.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.979+0000 7f9994fec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:01.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.979+0000 7f9994fec640 1 -- 172.21.15.170:0/2315319637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99800094d0 con 0x7f99900a8c00 2023-10-30T17:55:01.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.981+0000 7f9994fec640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900a8c00 0x7f99901d4980 secure :-1 s=READY pgs=5263 cs=0 l=1 rev1=1 crypto rx=0x7f9980033000 tx=0x7f99800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:01.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.981+0000 7f9985ffb640 10 client.?.objecter ms_handle_connect 0x7f99900a8c00 2023-10-30T17:55:01.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.981+0000 7f9985ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:01.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 1 -- 172.21.15.170:0/2315319637 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998003ac60 con 0x7f99900a8c00 2023-10-30T17:55:01.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:01.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:01.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 10 monclient: dump: 2023-10-30T17:55:01.031 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:01.031 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:01.032 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.032 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.032 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:01.032 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.032 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:01.033 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:01.033 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:01.033 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 10 monclient: _finish_auth 0 2023-10-30T17:55:01.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:55:01.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:01.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 1 -- 172.21.15.170:0/2315319637 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998003ade0 con 0x7f99900a8c00 2023-10-30T17:55:01.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9997a51640 5 monclient: authenticate success, global_id 21424 2023-10-30T17:55:01.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:01.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9997a51640 10 monclient: _renew_subs 2023-10-30T17:55:01.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 1 -- 172.21.15.170:0/2315319637 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9980044630 con 0x7f99900a8c00 2023-10-30T17:55:01.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9997a51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:01.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9997a51640 1 -- 172.21.15.170:0/2315319637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99901d4ec0 con 0x7f99900a8c00 2023-10-30T17:55:01.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:01.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:01.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9985ffb640 10 monclient: dump: 2023-10-30T17:55:01.036 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:01.036 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:01.036 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.036 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.036 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:01.037 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.037 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:01.037 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:01.037 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:01.037 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.037 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f99877fe640 4 set_mon_vals no callback set 2023-10-30T17:55:01.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9997a51640 10 client.21424.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:01.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9997a51640 10 monclient: _renew_subs 2023-10-30T17:55:01.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9997a51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:01.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9997a51640 1 -- 172.21.15.170:0/2315319637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9990139340 con 0x7f99900a8c00 2023-10-30T17:55:01.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.985+0000 7f9997a51640 1 librados: init done 2023-10-30T17:55:01.039 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.986+0000 7f9997a51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:01.039 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.986+0000 7f9997a51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:01.039 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.986+0000 7f9997a51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:01.039 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:00.986+0000 7f9997a51640 1 -- 172.21.15.170:0/2315319637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999004e3c0 con 0x7f99900a8c00 2023-10-30T17:55:01.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.009+0000 7f9985ffb640 1 -- 172.21.15.170:0/2315319637 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f998002f050 con 0x7f99900a8c00 2023-10-30T17:55:01.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.010+0000 7f9985ffb640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f997406c800 0x7f997406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:01.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.010+0000 7f99957ed640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f997406c800 0x7f997406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:01.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.010+0000 7f99957ed640 10 monclient: get_auth_request con 0x7f997406c800 auth_method 0 2023-10-30T17:55:01.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.011+0000 7f99957ed640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f997406c800 0x7f997406ecc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f998c009b60 tx=0x7f998c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:01.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.011+0000 7f9985ffb640 10 client.21424.objecter ms_handle_connect 0x7f997406c800 2023-10-30T17:55:01.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.238+0000 7f9985ffb640 1 -- 172.21.15.170:0/2315319637 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(913..913 src has 374..913) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f998007a4e0 con 0x7f99900a8c00 2023-10-30T17:55:01.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.238+0000 7f9985ffb640 10 client.21424.objecter ms_dispatch 0x7f9990000b90 osd_map(913..913 src has 374..913) v4 2023-10-30T17:55:01.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.238+0000 7f9985ffb640 3 client.21424.objecter handle_osd_map got epochs [913,913] > 0 2023-10-30T17:55:01.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.238+0000 7f9985ffb640 3 client.21424.objecter handle_osd_map decoding full epoch 913 2023-10-30T17:55:01.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.239+0000 7f9985ffb640 20 client.21424.objecter dump_active .. 0 homeless 2023-10-30T17:55:01.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.248+0000 7f9985ffb640 1 -- 172.21.15.170:0/2315319637 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9980076820 con 0x7f99900a8c00 2023-10-30T17:55:01.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.248+0000 7f9985ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:01.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.248+0000 7f9985ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:01.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.428+0000 7f9997a51640 1 -- 172.21.15.170:0/2315319637 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f999006d550 con 0x7f997406c800 2023-10-30T17:55:01.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.433+0000 7f9985ffb640 1 -- 172.21.15.170:0/2315319637 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151540 (secure 0 0 0) 0x7f999006d550 con 0x7f997406c800 2023-10-30T17:55:01.437 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:55:01.437 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-10-30T17:55:01.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 10 librados: watch_flush enter 2023-10-30T17:55:01.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 10 librados: watch_flush exit 2023-10-30T17:55:01.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 20 client.21424.objecter shutdown clearing up homeless session... 2023-10-30T17:55:01.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 10 client.21424.objecter successfully canceled tick 2023-10-30T17:55:01.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 -- 172.21.15.170:0/2315319637 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f997406c800 msgr2=0x7f997406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:01.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f997406c800 0x7f997406ecc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f998c009b60 tx=0x7f998c009190 comp rx=0 tx=0).stop 2023-10-30T17:55:01.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 10 monclient: shutdown 2023-10-30T17:55:01.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:01.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 -- 172.21.15.170:0/2315319637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900a8c00 msgr2=0x7f99901d4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:01.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900a8c00 0x7f99901d4980 secure :-1 s=READY pgs=5263 cs=0 l=1 rev1=1 crypto rx=0x7f9980033000 tx=0x7f99800046c0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 -- 172.21.15.170:0/2315319637 shutdown_connections 2023-10-30T17:55:01.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f997406c800 0x7f997406ecc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900a8c00 0x7f99901d4980 unknown :-1 s=CLOSED pgs=5263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99900b0330 0x7f9990138c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 --2- 172.21.15.170:0/2315319637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99900a7050 0x7f9990070b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 -- 172.21.15.170:0/2315319637 >> 172.21.15.170:0/2315319637 conn(0x7f99900b3560 msgr2=0x7f99900af500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:01.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 -- 172.21.15.170:0/2315319637 shutdown_connections 2023-10-30T17:55:01.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 -- 172.21.15.170:0/2315319637 wait complete. 2023-10-30T17:55:01.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.436+0000 7f9997a51640 1 librados: shutdown 2023-10-30T17:55:01.448 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2023-10-30T17:55:01.325774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":568424,"kb_used_data":60776,"kb_used_omap":12,"kb_used_meta":507507,"kb_avail":754406296,"statfs":{"total":773094113280,"available":772512047104,"internally_reserved":0,"allocated":62234624,"data_stored":34698846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":519687736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":375,"apply_latency_ms":375,"commit_latency_ns":375000000,"apply_latency_ns":375000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000948"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":856,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.965106+0000","last_change":"2023-10-30T17:33:37.648571+0000","last_active":"2023-10-30T17:54:56.965106+0000","last_peered":"2023-10-30T17:54:56.965106+0000","last_clean":"2023-10-30T17:54:56.965106+0000","last_became_active":"2023-10-30T17:33:37.643213+0000","last_became_peered":"2023-10-30T17:33:37.643213+0000","last_unstale":"2023-10-30T17:54:56.965106+0000","last_undegraded":"2023-10-30T17:54:56.965106+0000","last_fullsized":"2023-10-30T17:54:56.965106+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:43:30.718487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":853,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.252428+0000","last_change":"2023-10-30T17:33:37.836041+0000","last_active":"2023-10-30T17:54:59.252428+0000","last_peered":"2023-10-30T17:54:59.252428+0000","last_clean":"2023-10-30T17:54:59.252428+0000","last_became_active":"2023-10-30T17:33:37.827806+0000","last_became_peered":"2023-10-30T17:33:37.827806+0000","last_unstale":"2023-10-30T17:54:59.252428+0000","last_undegraded":"2023-10-30T17:54:59.252428+0000","last_fullsized":"2023-10-30T17:54:59.252428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:00.255976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":851,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.737796+0000","last_change":"2023-10-30T17:33:40.825119+0000","last_active":"2023-10-30T17:54:55.737796+0000","last_peered":"2023-10-30T17:54:55.737796+0000","last_clean":"2023-10-30T17:54:55.737796+0000","last_became_active":"2023-10-30T17:33:40.511287+0000","last_became_peered":"2023-10-30T17:33:40.511287+0000","last_unstale":"2023-10-30T17:54:55.737796+0000","last_undegraded":"2023-10-30T17:54:55.737796+0000","last_fullsized":"2023-10-30T17:54:55.737796+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:14:23.464432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":853,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.248681+0000","last_change":"2023-10-30T17:33:38.590694+0000","last_active":"2023-10-30T17:54:59.248681+0000","last_peered":"2023-10-30T17:54:59.248681+0000","last_clean":"2023-10-30T17:54:59.248681+0000","last_became_active":"2023-10-30T17:33:38.581457+0000","last_became_peered":"2023-10-30T17:33:38.581457+0000","last_unstale":"2023-10-30T17:54:59.248681+0000","last_undegraded":"2023-10-30T17:54:59.248681+0000","last_fullsized":"2023-10-30T17:54:59.248681+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:17.385325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":856,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.939866+0000","last_change":"2023-10-30T17:33:46.635360+0000","last_active":"2023-10-30T17:54:56.939866+0000","last_peered":"2023-10-30T17:54:56.939866+0000","last_clean":"2023-10-30T17:54:56.939866+0000","last_became_active":"2023-10-30T17:33:46.629442+0000","last_became_peered":"2023-10-30T17:33:46.629442+0000","last_unstale":"2023-10-30T17:54:56.939866+0000","last_undegraded":"2023-10-30T17:54:56.939866+0000","last_fullsized":"2023-10-30T17:54:56.939866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:46:06.082348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":852,"reported_epoch":909,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.139730+0000","last_change":"2023-10-30T17:33:39.141659+0000","last_active":"2023-10-30T17:54:57.139730+0000","last_peered":"2023-10-30T17:54:57.139730+0000","last_clean":"2023-10-30T17:54:57.139730+0000","last_became_active":"2023-10-30T17:33:39.135750+0000","last_became_peered":"2023-10-30T17:33:39.135750+0000","last_unstale":"2023-10-30T17:54:57.139730+0000","last_undegraded":"2023-10-30T17:54:57.139730+0000","last_fullsized":"2023-10-30T17:54:57.139730+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:27.698297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":853,"reported_epoch":909,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.207778+0000","last_change":"2023-10-30T17:33:51.011431+0000","last_active":"2023-10-30T17:54:57.207778+0000","last_peered":"2023-10-30T17:54:57.207778+0000","last_clean":"2023-10-30T17:54:57.207778+0000","last_became_active":"2023-10-30T17:33:51.001524+0000","last_became_peered":"2023-10-30T17:33:51.001524+0000","last_unstale":"2023-10-30T17:54:57.207778+0000","last_undegraded":"2023-10-30T17:54:57.207778+0000","last_fullsized":"2023-10-30T17:54:57.207778+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:41.602832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.027235+0000","last_change":"2023-10-30T17:33:58.879816+0000","last_active":"2023-10-30T17:54:58.027235+0000","last_peered":"2023-10-30T17:54:58.027235+0000","last_clean":"2023-10-30T17:54:58.027235+0000","last_became_active":"2023-10-30T17:33:58.874661+0000","last_became_peered":"2023-10-30T17:33:58.874661+0000","last_unstale":"2023-10-30T17:54:58.027235+0000","last_undegraded":"2023-10-30T17:54:58.027235+0000","last_fullsized":"2023-10-30T17:54:58.027235+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:04:24.038125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":849,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.993781+0000","last_change":"2023-10-30T17:33:39.104029+0000","last_active":"2023-10-30T17:54:58.993781+0000","last_peered":"2023-10-30T17:54:58.993781+0000","last_clean":"2023-10-30T17:54:58.993781+0000","last_became_active":"2023-10-30T17:33:39.086390+0000","last_became_peered":"2023-10-30T17:33:39.086390+0000","last_unstale":"2023-10-30T17:54:58.993781+0000","last_undegraded":"2023-10-30T17:54:58.993781+0000","last_fullsized":"2023-10-30T17:54:58.993781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:52.540858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":853,"reported_epoch":909,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.186394+0000","last_change":"2023-10-30T17:33:40.038786+0000","last_active":"2023-10-30T17:54:57.186394+0000","last_peered":"2023-10-30T17:54:57.186394+0000","last_clean":"2023-10-30T17:54:57.186394+0000","last_became_active":"2023-10-30T17:33:39.965244+0000","last_became_peered":"2023-10-30T17:33:39.965244+0000","last_unstale":"2023-10-30T17:54:57.186394+0000","last_undegraded":"2023-10-30T17:54:57.186394+0000","last_fullsized":"2023-10-30T17:54:57.186394+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:17:21.402055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":857,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.954339+0000","last_change":"2023-10-30T17:33:46.567515+0000","last_active":"2023-10-30T17:54:56.954339+0000","last_peered":"2023-10-30T17:54:56.954339+0000","last_clean":"2023-10-30T17:54:56.954339+0000","last_became_active":"2023-10-30T17:33:46.562080+0000","last_became_peered":"2023-10-30T17:33:46.562080+0000","last_unstale":"2023-10-30T17:54:56.954339+0000","last_undegraded":"2023-10-30T17:54:56.954339+0000","last_fullsized":"2023-10-30T17:54:56.954339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:42:52.518433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.979403+0000","last_change":"2023-10-30T17:33:40.831046+0000","last_active":"2023-10-30T17:54:57.979403+0000","last_peered":"2023-10-30T17:54:57.979403+0000","last_clean":"2023-10-30T17:54:57.979403+0000","last_became_active":"2023-10-30T17:33:40.798989+0000","last_became_peered":"2023-10-30T17:33:40.798989+0000","last_unstale":"2023-10-30T17:54:57.979403+0000","last_undegraded":"2023-10-30T17:54:57.979403+0000","last_fullsized":"2023-10-30T17:54:57.979403+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:35.578143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":853,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.023321+0000","last_change":"2023-10-30T17:32:35.372482+0000","last_active":"2023-10-30T17:54:59.023321+0000","last_peered":"2023-10-30T17:54:59.023321+0000","last_clean":"2023-10-30T17:54:59.023321+0000","last_became_active":"2023-10-30T17:32:35.367487+0000","last_became_peered":"2023-10-30T17:32:35.367487+0000","last_unstale":"2023-10-30T17:54:59.023321+0000","last_undegraded":"2023-10-30T17:54:59.023321+0000","last_fullsized":"2023-10-30T17:54:59.023321+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:45:03.846674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":858,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.920758+0000","last_change":"2023-10-30T17:33:38.291772+0000","last_active":"2023-10-30T17:54:56.920758+0000","last_peered":"2023-10-30T17:54:56.920758+0000","last_clean":"2023-10-30T17:54:56.920758+0000","last_became_active":"2023-10-30T17:33:38.284317+0000","last_became_peered":"2023-10-30T17:33:38.284317+0000","last_unstale":"2023-10-30T17:54:56.920758+0000","last_undegraded":"2023-10-30T17:54:56.920758+0000","last_fullsized":"2023-10-30T17:54:56.920758+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:02.583403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":863,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.945069+0000","last_change":"2023-10-30T17:32:34.589393+0000","last_active":"2023-10-30T17:54:56.945069+0000","last_peered":"2023-10-30T17:54:56.945069+0000","last_clean":"2023-10-30T17:54:56.945069+0000","last_became_active":"2023-10-30T17:32:34.557021+0000","last_became_peered":"2023-10-30T17:32:34.557021+0000","last_unstale":"2023-10-30T17:54:56.945069+0000","last_undegraded":"2023-10-30T17:54:56.945069+0000","last_fullsized":"2023-10-30T17:54:56.945069+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:05.352859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"47'7","reported_seq":863,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.992239+0000","last_change":"2023-10-30T17:33:39.222547+0000","last_active":"2023-10-30T17:54:58.992239+0000","last_peered":"2023-10-30T17:54:58.992239+0000","last_clean":"2023-10-30T17:54:58.992239+0000","last_became_active":"2023-10-30T17:33:39.211129+0000","last_became_peered":"2023-10-30T17:33:39.211129+0000","last_unstale":"2023-10-30T17:54:58.992239+0000","last_undegraded":"2023-10-30T17:54:58.992239+0000","last_fullsized":"2023-10-30T17:54:58.992239+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:49.479747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.008700+0000","last_change":"2023-10-30T17:32:35.362652+0000","last_active":"2023-10-30T17:54:59.008700+0000","last_peered":"2023-10-30T17:54:59.008700+0000","last_clean":"2023-10-30T17:54:59.008700+0000","last_became_active":"2023-10-30T17:32:35.357489+0000","last_became_peered":"2023-10-30T17:32:35.357489+0000","last_unstale":"2023-10-30T17:54:59.008700+0000","last_undegraded":"2023-10-30T17:54:59.008700+0000","last_fullsized":"2023-10-30T17:54:59.008700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:21:36.073054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":855,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.609324+0000","last_change":"2023-10-30T17:33:40.644802+0000","last_active":"2023-10-30T17:54:54.609324+0000","last_peered":"2023-10-30T17:54:54.609324+0000","last_clean":"2023-10-30T17:54:54.609324+0000","last_became_active":"2023-10-30T17:33:40.584323+0000","last_became_peered":"2023-10-30T17:33:40.584323+0000","last_unstale":"2023-10-30T17:54:54.609324+0000","last_undegraded":"2023-10-30T17:54:54.609324+0000","last_fullsized":"2023-10-30T17:54:54.609324+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:41:03.792929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":860,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.906532+0000","last_change":"2023-10-30T17:32:34.658442+0000","last_active":"2023-10-30T17:54:55.906532+0000","last_peered":"2023-10-30T17:54:55.906532+0000","last_clean":"2023-10-30T17:54:55.906532+0000","last_became_active":"2023-10-30T17:32:34.631494+0000","last_became_peered":"2023-10-30T17:32:34.631494+0000","last_unstale":"2023-10-30T17:54:55.906532+0000","last_undegraded":"2023-10-30T17:54:55.906532+0000","last_fullsized":"2023-10-30T17:54:55.906532+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:36:36.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":858,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.956509+0000","last_change":"2023-10-30T17:33:46.593988+0000","last_active":"2023-10-30T17:54:56.956509+0000","last_peered":"2023-10-30T17:54:56.956509+0000","last_clean":"2023-10-30T17:54:56.956509+0000","last_became_active":"2023-10-30T17:33:46.588555+0000","last_became_peered":"2023-10-30T17:33:46.588555+0000","last_unstale":"2023-10-30T17:54:56.956509+0000","last_undegraded":"2023-10-30T17:54:56.956509+0000","last_fullsized":"2023-10-30T17:54:56.956509+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:57.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.998017+0000","last_change":"2023-10-30T17:32:35.388556+0000","last_active":"2023-10-30T17:54:58.998017+0000","last_peered":"2023-10-30T17:54:58.998017+0000","last_clean":"2023-10-30T17:54:58.998017+0000","last_became_active":"2023-10-30T17:32:35.383414+0000","last_became_peered":"2023-10-30T17:32:35.383414+0000","last_unstale":"2023-10-30T17:54:58.998017+0000","last_undegraded":"2023-10-30T17:54:58.998017+0000","last_fullsized":"2023-10-30T17:54:58.998017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:46.189212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":871,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.595206+0000","last_change":"2023-10-30T17:33:38.185755+0000","last_active":"2023-10-30T17:54:54.595206+0000","last_peered":"2023-10-30T17:54:54.595206+0000","last_clean":"2023-10-30T17:54:54.595206+0000","last_became_active":"2023-10-30T17:33:38.176989+0000","last_became_peered":"2023-10-30T17:33:38.176989+0000","last_unstale":"2023-10-30T17:54:54.595206+0000","last_undegraded":"2023-10-30T17:54:54.595206+0000","last_fullsized":"2023-10-30T17:54:54.595206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:25:37.689974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":878,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.580413+0000","last_change":"2023-10-30T17:32:34.318569+0000","last_active":"2023-10-30T17:54:54.580413+0000","last_peered":"2023-10-30T17:54:54.580413+0000","last_clean":"2023-10-30T17:54:54.580413+0000","last_became_active":"2023-10-30T17:32:34.287031+0000","last_became_peered":"2023-10-30T17:32:34.287031+0000","last_unstale":"2023-10-30T17:54:54.580413+0000","last_undegraded":"2023-10-30T17:54:54.580413+0000","last_fullsized":"2023-10-30T17:54:54.580413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:55:59.080654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":870,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.610146+0000","last_change":"2023-10-30T17:33:41.123654+0000","last_active":"2023-10-30T17:54:54.610146+0000","last_peered":"2023-10-30T17:54:54.610146+0000","last_clean":"2023-10-30T17:54:54.610146+0000","last_became_active":"2023-10-30T17:33:41.116885+0000","last_became_peered":"2023-10-30T17:33:41.116885+0000","last_unstale":"2023-10-30T17:54:54.610146+0000","last_undegraded":"2023-10-30T17:54:54.610146+0000","last_fullsized":"2023-10-30T17:54:54.610146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:51:37.700398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.996395+0000","last_change":"2023-10-30T17:32:35.307241+0000","last_active":"2023-10-30T17:54:58.996395+0000","last_peered":"2023-10-30T17:54:58.996395+0000","last_clean":"2023-10-30T17:54:58.996395+0000","last_became_active":"2023-10-30T17:32:35.277163+0000","last_became_peered":"2023-10-30T17:32:35.277163+0000","last_unstale":"2023-10-30T17:54:58.996395+0000","last_undegraded":"2023-10-30T17:54:58.996395+0000","last_fullsized":"2023-10-30T17:54:58.996395+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:02.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":848,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.025231+0000","last_change":"2023-10-30T17:33:43.745646+0000","last_active":"2023-10-30T17:54:59.025231+0000","last_peered":"2023-10-30T17:54:59.025231+0000","last_clean":"2023-10-30T17:54:59.025231+0000","last_became_active":"2023-10-30T17:33:43.736021+0000","last_became_peered":"2023-10-30T17:33:43.736021+0000","last_unstale":"2023-10-30T17:54:59.025231+0000","last_undegraded":"2023-10-30T17:54:59.025231+0000","last_fullsized":"2023-10-30T17:54:59.025231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:29.825853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.021371+0000","last_change":"2023-10-30T17:32:35.628758+0000","last_active":"2023-10-30T17:54:59.021371+0000","last_peered":"2023-10-30T17:54:59.021371+0000","last_clean":"2023-10-30T17:54:59.021371+0000","last_became_active":"2023-10-30T17:32:35.623395+0000","last_became_peered":"2023-10-30T17:32:35.623395+0000","last_unstale":"2023-10-30T17:54:59.021371+0000","last_undegraded":"2023-10-30T17:54:59.021371+0000","last_fullsized":"2023-10-30T17:54:59.021371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:47:08.984576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":966,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.254775+0000","last_change":"2023-10-30T17:32:28.719757+0000","last_active":"2023-10-30T17:54:59.254775+0000","last_peered":"2023-10-30T17:54:59.254775+0000","last_clean":"2023-10-30T17:54:59.254775+0000","last_became_active":"2023-10-30T17:32:28.686958+0000","last_became_peered":"2023-10-30T17:32:28.686958+0000","last_unstale":"2023-10-30T17:54:59.254775+0000","last_undegraded":"2023-10-30T17:54:59.254775+0000","last_fullsized":"2023-10-30T17:54:59.254775+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:25.778908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:25.778908+0000","last_clean_scrub_stamp":"2023-10-30T17:32:25.778908+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:41:21.329138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":855,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.630478+0000","last_change":"2023-10-30T17:33:40.638676+0000","last_active":"2023-10-30T17:54:54.630478+0000","last_peered":"2023-10-30T17:54:54.630478+0000","last_clean":"2023-10-30T17:54:54.630478+0000","last_became_active":"2023-10-30T17:33:40.602436+0000","last_became_peered":"2023-10-30T17:33:40.602436+0000","last_unstale":"2023-10-30T17:54:54.630478+0000","last_undegraded":"2023-10-30T17:54:54.630478+0000","last_fullsized":"2023-10-30T17:54:54.630478+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:41.628722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":849,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.013211+0000","last_change":"2023-10-30T17:33:38.623594+0000","last_active":"2023-10-30T17:54:59.013211+0000","last_peered":"2023-10-30T17:54:59.013211+0000","last_clean":"2023-10-30T17:54:59.013211+0000","last_became_active":"2023-10-30T17:33:38.616054+0000","last_became_peered":"2023-10-30T17:33:38.616054+0000","last_unstale":"2023-10-30T17:54:59.013211+0000","last_undegraded":"2023-10-30T17:54:59.013211+0000","last_fullsized":"2023-10-30T17:54:59.013211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:05:44.520215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":854,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.272308+0000","last_change":"2023-10-30T17:33:40.742636+0000","last_active":"2023-10-30T17:54:59.272308+0000","last_peered":"2023-10-30T17:54:59.272308+0000","last_clean":"2023-10-30T17:54:59.272308+0000","last_became_active":"2023-10-30T17:33:40.735511+0000","last_became_peered":"2023-10-30T17:33:40.735511+0000","last_unstale":"2023-10-30T17:54:59.272308+0000","last_undegraded":"2023-10-30T17:54:59.272308+0000","last_fullsized":"2023-10-30T17:54:59.272308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:30.315904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":854,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.905041+0000","last_change":"2023-10-30T17:33:39.276657+0000","last_active":"2023-10-30T17:54:55.905041+0000","last_peered":"2023-10-30T17:54:55.905041+0000","last_clean":"2023-10-30T17:54:55.905041+0000","last_became_active":"2023-10-30T17:33:39.228561+0000","last_became_peered":"2023-10-30T17:33:39.228561+0000","last_unstale":"2023-10-30T17:54:55.905041+0000","last_undegraded":"2023-10-30T17:54:55.905041+0000","last_fullsized":"2023-10-30T17:54:55.905041+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:01.276873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":870,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.597303+0000","last_change":"2023-10-30T17:33:41.611205+0000","last_active":"2023-10-30T17:54:54.597303+0000","last_peered":"2023-10-30T17:54:54.597303+0000","last_clean":"2023-10-30T17:54:54.597303+0000","last_became_active":"2023-10-30T17:33:41.605208+0000","last_became_peered":"2023-10-30T17:33:41.605208+0000","last_unstale":"2023-10-30T17:54:54.597303+0000","last_undegraded":"2023-10-30T17:54:54.597303+0000","last_fullsized":"2023-10-30T17:54:54.597303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:09.587584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.980989+0000","last_change":"2023-10-30T17:33:44.441890+0000","last_active":"2023-10-30T17:54:57.980989+0000","last_peered":"2023-10-30T17:54:57.980989+0000","last_clean":"2023-10-30T17:54:57.980989+0000","last_became_active":"2023-10-30T17:33:44.434104+0000","last_became_peered":"2023-10-30T17:33:44.434104+0000","last_unstale":"2023-10-30T17:54:57.980989+0000","last_undegraded":"2023-10-30T17:54:57.980989+0000","last_fullsized":"2023-10-30T17:54:57.980989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:11.365044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":855,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.899091+0000","last_change":"2023-10-30T17:33:43.330759+0000","last_active":"2023-10-30T17:54:55.899091+0000","last_peered":"2023-10-30T17:54:55.899091+0000","last_clean":"2023-10-30T17:54:55.899091+0000","last_became_active":"2023-10-30T17:33:43.322860+0000","last_became_peered":"2023-10-30T17:33:43.322860+0000","last_unstale":"2023-10-30T17:54:55.899091+0000","last_undegraded":"2023-10-30T17:54:55.899091+0000","last_fullsized":"2023-10-30T17:54:55.899091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:56:01.616064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.020934+0000","last_change":"2023-10-30T17:33:53.140054+0000","last_active":"2023-10-30T17:54:58.020934+0000","last_peered":"2023-10-30T17:54:58.020934+0000","last_clean":"2023-10-30T17:54:58.020934+0000","last_became_active":"2023-10-30T17:33:53.130615+0000","last_became_peered":"2023-10-30T17:33:53.130615+0000","last_unstale":"2023-10-30T17:54:58.020934+0000","last_undegraded":"2023-10-30T17:54:58.020934+0000","last_fullsized":"2023-10-30T17:54:58.020934+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:11.025225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":854,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.244856+0000","last_change":"2023-10-30T17:33:37.571189+0000","last_active":"2023-10-30T17:54:59.244856+0000","last_peered":"2023-10-30T17:54:59.244856+0000","last_clean":"2023-10-30T17:54:59.244856+0000","last_became_active":"2023-10-30T17:33:37.540749+0000","last_became_peered":"2023-10-30T17:33:37.540749+0000","last_unstale":"2023-10-30T17:54:59.244856+0000","last_undegraded":"2023-10-30T17:54:59.244856+0000","last_fullsized":"2023-10-30T17:54:59.244856+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:42.432458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":848,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.016385+0000","last_change":"2023-10-30T17:33:58.392321+0000","last_active":"2023-10-30T17:54:59.016385+0000","last_peered":"2023-10-30T17:54:59.016385+0000","last_clean":"2023-10-30T17:54:59.016385+0000","last_became_active":"2023-10-30T17:33:58.383320+0000","last_became_peered":"2023-10-30T17:33:58.383320+0000","last_unstale":"2023-10-30T17:54:59.016385+0000","last_undegraded":"2023-10-30T17:54:59.016385+0000","last_fullsized":"2023-10-30T17:54:59.016385+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:05.698087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":855,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.949358+0000","last_change":"2023-10-30T17:33:38.595237+0000","last_active":"2023-10-30T17:54:56.949358+0000","last_peered":"2023-10-30T17:54:56.949358+0000","last_clean":"2023-10-30T17:54:56.949358+0000","last_became_active":"2023-10-30T17:33:38.589479+0000","last_became_peered":"2023-10-30T17:33:38.589479+0000","last_unstale":"2023-10-30T17:54:56.949358+0000","last_undegraded":"2023-10-30T17:54:56.949358+0000","last_fullsized":"2023-10-30T17:54:56.949358+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:17:44.312113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":857,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.959591+0000","last_change":"2023-10-30T17:33:40.505225+0000","last_active":"2023-10-30T17:54:56.959591+0000","last_peered":"2023-10-30T17:54:56.959591+0000","last_clean":"2023-10-30T17:54:56.959591+0000","last_became_active":"2023-10-30T17:33:40.476315+0000","last_became_peered":"2023-10-30T17:33:40.476315+0000","last_unstale":"2023-10-30T17:54:56.959591+0000","last_undegraded":"2023-10-30T17:54:56.959591+0000","last_fullsized":"2023-10-30T17:54:56.959591+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:57:02.608925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":848,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.006691+0000","last_change":"2023-10-30T17:33:38.803571+0000","last_active":"2023-10-30T17:54:59.006691+0000","last_peered":"2023-10-30T17:54:59.006691+0000","last_clean":"2023-10-30T17:54:59.006691+0000","last_became_active":"2023-10-30T17:33:38.795801+0000","last_became_peered":"2023-10-30T17:33:38.795801+0000","last_unstale":"2023-10-30T17:54:59.006691+0000","last_undegraded":"2023-10-30T17:54:59.006691+0000","last_fullsized":"2023-10-30T17:54:59.006691+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:40:52.889892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":17,"seq":73014444302,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60904,"kb_used_data":7512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310936,"statfs":{"total":96636764160,"available":96574398464,"internally_reserved":0,"allocated":7692288,"data_stored":4265037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":7.8239999999999998,"5min":14.673999999999999,"15min":20.202999999999999},"min":{"1min":4.2460000000000004,"5min":4.2460000000000004,"15min":4.2460000000000004},"max":{"1min":15.343,"5min":217.97399999999999,"15min":296.49400000000003},"last":6.1900000000000004},{"interface":"front","average":{"1min":13.112,"5min":16.077000000000002,"15min":21.355},"min":{"1min":7.6120000000000001,"5min":5.6189999999999998,"15min":4.6299999999999999},"max":{"1min":31.678999999999998,"5min":134.833,"15min":276.49799999999999},"last":8.9879999999999995}]},{"osd":2,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":16.689,"5min":19.114999999999998,"15min":26.032},"min":{"1min":6.319,"5min":5.915,"15min":5.915},"max":{"1min":101.64100000000001,"5min":218.22399999999999,"15min":591.02499999999998},"last":11.734},{"interface":"front","average":{"1min":19.178999999999998,"5min":20.963000000000001,"15min":25.277000000000001},"min":{"1min":9.0850000000000009,"5min":7.6520000000000001,"15min":6.4930000000000003},"max":{"1min":103.506,"5min":228.36500000000001,"15min":259.52300000000002},"last":10.427}]},{"osd":3,"last update":"Mon Oct 30 17:53:58 2023","interfaces":[{"interface":"back","average":{"1min":23.952999999999999,"5min":78.492000000000004,"15min":43.039000000000001},"min":{"1min":6.9329999999999998,"5min":6.6879999999999997,"15min":5.8099999999999996},"max":{"1min":282.685,"5min":5036.1790000000001,"15min":5036.1790000000001},"last":9.7240000000000002},{"interface":"front","average":{"1min":24.942,"5min":81.421000000000006,"15min":47.087000000000003},"min":{"1min":8.4399999999999995,"5min":7.2770000000000001,"15min":6.4909999999999997},"max":{"1min":190.00899999999999,"5min":5057.1229999999996,"15min":5057.1229999999996},"last":13.465999999999999}]},{"osd":4,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":24.963000000000001,"5min":22.420999999999999,"15min":22.024000000000001},"min":{"1min":7.3259999999999996,"5min":7.3259999999999996,"15min":6.3689999999999998},"max":{"1min":159.94399999999999,"5min":204.91,"15min":281.16399999999999},"last":16.094000000000001},{"interface":"front","average":{"1min":29.393999999999998,"5min":26.954000000000001,"15min":27.579000000000001},"min":{"1min":9.0530000000000008,"5min":9.0530000000000008,"15min":5.8879999999999999},"max":{"1min":190.24199999999999,"5min":218.85300000000001,"15min":297.57299999999998},"last":14.125}]},{"osd":5,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":19.146999999999998,"5min":25.224,"15min":28.202999999999999},"min":{"1min":8.6660000000000004,"5min":8.4960000000000004,"15min":7.9370000000000003},"max":{"1min":44.509,"5min":260.81099999999998,"15min":260.81099999999998},"last":18.850000000000001},{"interface":"front","average":{"1min":15.294,"5min":22.741,"15min":25.870000000000001},"min":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.1189999999999998},"max":{"1min":32.457999999999998,"5min":236.83500000000001,"15min":236.83500000000001},"last":16.760999999999999}]},{"osd":6,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":19.766999999999999,"5min":22.66,"15min":27.329000000000001},"min":{"1min":10.051,"5min":9.968,"15min":8.4019999999999992},"max":{"1min":37.848999999999997,"5min":125.158,"15min":292.47500000000002},"last":17.888000000000002},{"interface":"front","average":{"1min":20.893000000000001,"5min":26.652000000000001,"15min":28.780000000000001},"min":{"1min":11.971,"5min":6.8449999999999998,"15min":6.8449999999999998},"max":{"1min":62.518000000000001,"5min":189.01599999999999,"15min":273.64999999999998},"last":15.422000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":22.042999999999999,"5min":25.934999999999999,"15min":29.561},"min":{"1min":12.084,"5min":10.622999999999999,"15min":9.157},"max":{"1min":60.759,"5min":204.12200000000001,"15min":262.74200000000002},"last":12.778},{"interface":"front","average":{"1min":26.411999999999999,"5min":31.727,"15min":33.009999999999998},"min":{"1min":13.321,"5min":12.188000000000001,"15min":10.122999999999999},"max":{"1min":100.953,"5min":219.833,"15min":264.81},"last":19.789000000000001}]}]},{"osd":3,"up_from":16,"seq":68719476998,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83092,"kb_used_data":7492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288748,"statfs":{"total":96636764160,"available":96551677952,"internally_reserved":0,"allocated":7671808,"data_stored":4216833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":9.1400000000000006,"5min":16.59,"15min":16.803999999999998},"min":{"1min":4.734,"5min":4.3869999999999996,"15min":4.3869999999999996},"max":{"1min":19.263000000000002,"5min":203.85499999999999,"15min":271.12799999999999},"last":9.2040000000000006},{"interface":"front","average":{"1min":12.346,"5min":17.565999999999999,"15min":18.774000000000001},"min":{"1min":5.5490000000000004,"5min":4.8449999999999998,"15min":4.367},"max":{"1min":36.253,"5min":176.83500000000001,"15min":364.05399999999997},"last":11.433999999999999}]},{"osd":1,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":19.100000000000001,"5min":22.856999999999999,"15min":24.716000000000001},"min":{"1min":7.3470000000000004,"5min":6.6589999999999998,"15min":5.5380000000000003},"max":{"1min":74.484999999999999,"5min":178.04300000000001,"15min":314.19099999999997},"last":19.881},{"interface":"front","average":{"1min":24.044,"5min":23.404,"15min":23.681000000000001},"min":{"1min":6.5609999999999999,"5min":5.9290000000000003,"15min":5.609},"max":{"1min":159.42099999999999,"5min":210.381,"15min":240.64500000000001},"last":13.476000000000001}]},{"osd":2,"last update":"Mon Oct 30 17:54:19 2023","interfaces":[{"interface":"back","average":{"1min":34.308999999999997,"5min":27.562000000000001,"15min":24.027999999999999},"min":{"1min":7.2969999999999997,"5min":7.2489999999999997,"15min":5.234},"max":{"1min":251.941,"5min":251.941,"15min":397.50099999999998},"last":17.768999999999998},{"interface":"front","average":{"1min":19.899000000000001,"5min":22.161999999999999,"15min":25.760999999999999},"min":{"1min":7.1539999999999999,"5min":6.54,"15min":6.54},"max":{"1min":85.495000000000005,"5min":147.50399999999999,"15min":219.65000000000001},"last":10.699}]},{"osd":4,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":22.57,"5min":33.298000000000002,"15min":27.831},"min":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":5.8140000000000001},"max":{"1min":164.642,"5min":318.92200000000003,"15min":400.13999999999999},"last":12.052},{"interface":"front","average":{"1min":17.919,"5min":25.715,"15min":22.006},"min":{"1min":8.3650000000000002,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":83.346999999999994,"5min":268.01600000000002,"15min":325.846},"last":18.446999999999999}]},{"osd":5,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":20.472000000000001,"5min":21.146999999999998,"15min":23.655000000000001},"min":{"1min":9.2029999999999994,"5min":7.7569999999999997,"15min":7.7569999999999997},"max":{"1min":84.305000000000007,"5min":149.61600000000001,"15min":221.821},"last":16.879000000000001},{"interface":"front","average":{"1min":30.245000000000001,"5min":27.282,"15min":28.794},"min":{"1min":9.75,"5min":7.2009999999999996,"15min":7.2009999999999996},"max":{"1min":225.15299999999999,"5min":225.15299999999999,"15min":372.13999999999999},"last":15.724}]},{"osd":6,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":32.417999999999999,"5min":30.308,"15min":28.475000000000001},"min":{"1min":10.678000000000001,"5min":7.9589999999999996,"15min":7.7439999999999998},"max":{"1min":272.536,"5min":272.536,"15min":360.26999999999998},"last":58.084000000000003},{"interface":"front","average":{"1min":23.282,"5min":23.863,"15min":25.344000000000001},"min":{"1min":11.023999999999999,"5min":9.3040000000000003,"15min":7.524},"max":{"1min":113.96599999999999,"5min":162.61799999999999,"15min":336.68700000000001},"last":97.793000000000006}]},{"osd":7,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":27.902999999999999,"5min":31.673999999999999,"15min":29.501000000000001},"min":{"1min":12.977,"5min":9.8399999999999999,"15min":9.7059999999999995},"max":{"1min":178.43100000000001,"5min":196.05699999999999,"15min":281.15699999999998},"last":55.932000000000002},{"interface":"front","average":{"1min":31.780999999999999,"5min":30.504999999999999,"15min":29.385999999999999},"min":{"1min":11.692,"5min":10.587999999999999,"15min":10.528},"max":{"1min":177.44900000000001,"5min":182.07499999999999,"15min":233.489},"last":31.356999999999999}]}]},{"osd":1,"up_from":16,"seq":68719477009,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61300,"kb_used_data":7904,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310540,"statfs":{"total":96636764160,"available":96573992960,"internally_reserved":0,"allocated":8093696,"data_stored":4628450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":10.568,"5min":13.151999999999999,"15min":17.620999999999999},"min":{"1min":5.1379999999999999,"5min":4.8099999999999996,"15min":3.8279999999999998},"max":{"1min":26.812000000000001,"5min":162.364,"15min":313.69400000000002},"last":11.885999999999999},{"interface":"front","average":{"1min":13.189,"5min":13.273999999999999,"15min":17.713000000000001},"min":{"1min":8.4429999999999996,"5min":5.4560000000000004,"15min":4.9089999999999998},"max":{"1min":21.195,"5min":59.277000000000001,"15min":209.62799999999999},"last":8.4979999999999993}]},{"osd":2,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":17.059999999999999,"5min":16.346,"15min":21.375},"min":{"1min":8.5259999999999998,"5min":6.2190000000000003,"15min":6.0279999999999996},"max":{"1min":84.930999999999997,"5min":95.995000000000005,"15min":267.83499999999998},"last":14.233000000000001},{"interface":"front","average":{"1min":19.794,"5min":18.084,"15min":20.032},"min":{"1min":11.779999999999999,"5min":7.0270000000000001,"15min":5.8550000000000004},"max":{"1min":89.875,"5min":89.875,"15min":178.44},"last":26.521000000000001}]},{"osd":3,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":10.504,"5min":246.75800000000001,"15min":96.697000000000003},"min":{"1min":6.1180000000000003,"5min":6.1050000000000004,"15min":5.2990000000000004},"max":{"1min":31.059000000000001,"5min":8788.2559999999994,"15min":8788.2559999999994},"last":8.9199999999999999},{"interface":"front","average":{"1min":17.736999999999998,"5min":247.22,"15min":98.808000000000007},"min":{"1min":11.208,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":29.475999999999999,"5min":8677.7369999999992,"15min":8677.7369999999992},"last":29.404}]},{"osd":4,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":15.772,"5min":17.181999999999999,"15min":23.558},"min":{"1min":7.7359999999999998,"5min":6.8220000000000001,"15min":6.8220000000000001},"max":{"1min":91.501999999999995,"5min":91.501999999999995,"15min":235.125},"last":9.5410000000000004},{"interface":"front","average":{"1min":20.797999999999998,"5min":21.347000000000001,"15min":23.934000000000001},"min":{"1min":10.278,"5min":7.673,"15min":7.1829999999999998},"max":{"1min":92.772000000000006,"5min":92.772000000000006,"15min":334.19900000000001},"last":13.321}]},{"osd":5,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":15.865,"5min":19.510000000000002,"15min":27.055},"min":{"1min":9.0389999999999997,"5min":9.0389999999999997,"15min":7.6360000000000001},"max":{"1min":43.468000000000004,"5min":89.837000000000003,"15min":220.65799999999999},"last":28.439},{"interface":"front","average":{"1min":21.504999999999999,"5min":21.619,"15min":31.059999999999999},"min":{"1min":14.56,"5min":8.0869999999999997,"15min":8.0869999999999997},"max":{"1min":50.332000000000001,"5min":134.42599999999999,"15min":262.61799999999999},"last":32.908000000000001}]},{"osd":6,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":10.253,"5min":15.454000000000001,"15min":24.402000000000001},"min":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":6.8769999999999998},"max":{"1min":20.927,"5min":200.369,"15min":249.25299999999999},"last":16.367000000000001},{"interface":"front","average":{"1min":17.808,"5min":20.117999999999999,"15min":21.41},"min":{"1min":10.965,"5min":8.8610000000000007,"15min":6.2880000000000003},"max":{"1min":28.323,"5min":201.756,"15min":259.03899999999999},"last":31.189}]},{"osd":7,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":17.879000000000001,"5min":21.974,"15min":27.113},"min":{"1min":11.935,"5min":9.8200000000000003,"15min":9.4269999999999996},"max":{"1min":29.427,"5min":100.73999999999999,"15min":200.62100000000001},"last":25.783000000000001},{"interface":"front","average":{"1min":21.350999999999999,"5min":26.129999999999999,"15min":30.661000000000001},"min":{"1min":16.481000000000002,"5min":10.682,"15min":10.115},"max":{"1min":36.606999999999999,"5min":203.929,"15min":205.61799999999999},"last":32.268000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509712,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60844,"kb_used_data":7460,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310996,"statfs":{"total":96636764160,"available":96574459904,"internally_reserved":0,"allocated":7639040,"data_stored":4202947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":14.316000000000001,"5min":13.782,"15min":18.59},"min":{"1min":5.3460000000000001,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":106.839,"5min":106.839,"15min":334.36900000000003},"last":6.2960000000000003},{"interface":"front","average":{"1min":18.443000000000001,"5min":17.509,"15min":21.878},"min":{"1min":7.7990000000000004,"5min":7.1319999999999997,"15min":6.7489999999999997},"max":{"1min":111.07599999999999,"5min":111.07599999999999,"15min":283.46899999999999},"last":11.449}]},{"osd":1,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":25.234000000000002,"5min":23.196999999999999,"15min":24.32},"min":{"1min":9.1259999999999994,"5min":6.1539999999999999,"15min":4.7830000000000004},"max":{"1min":122.607,"5min":194.499,"15min":273.93599999999998},"last":12.382999999999999},{"interface":"front","average":{"1min":18.93,"5min":14.327999999999999,"15min":15.942},"min":{"1min":5.0069999999999997,"5min":4.9020000000000001,"15min":4.1890000000000001},"max":{"1min":109.767,"5min":109.767,"15min":152.87299999999999},"last":7.9729999999999999}]},{"osd":3,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":12.736000000000001,"5min":219.62799999999999,"15min":85.793999999999997},"min":{"1min":5.6429999999999998,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":62.838000000000001,"5min":8749.8639999999996,"15min":8749.8639999999996},"last":108.666},{"interface":"front","average":{"1min":19.050999999999998,"5min":225.965,"15min":91.936000000000007},"min":{"1min":9.2469999999999999,"5min":7.7859999999999996,"15min":6.6799999999999997},"max":{"1min":75.923000000000002,"5min":8752.7450000000008,"15min":8752.7450000000008},"last":109.779}]},{"osd":4,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":24.823,"5min":22.719000000000001,"15min":28.533999999999999},"min":{"1min":8.8420000000000005,"5min":8.5990000000000002,"15min":8.5990000000000002},"max":{"1min":115.70999999999999,"5min":124.502,"15min":301.34399999999999},"last":14.112},{"interface":"front","average":{"1min":27.699999999999999,"5min":27.585000000000001,"15min":32.787999999999997},"min":{"1min":11.974,"5min":11.974,"15min":11.824},"max":{"1min":117.285,"5min":134.44800000000001,"15min":300.68900000000002},"last":15.726000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":21.994,"5min":19.698,"15min":29.402999999999999},"min":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":124.914,"5min":198.46700000000001,"15min":318.01900000000001},"last":25.18},{"interface":"front","average":{"1min":21.603999999999999,"5min":18.942,"15min":24.146000000000001},"min":{"1min":10.16,"5min":7.5960000000000001,"15min":5.8920000000000003},"max":{"1min":122.861,"5min":122.861,"15min":325.87099999999998},"last":107.361}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":19.571000000000002,"5min":18.742000000000001,"15min":20.893000000000001},"min":{"1min":8.9290000000000003,"5min":8.7420000000000009,"15min":6.3979999999999997},"max":{"1min":70.927999999999997,"5min":70.927999999999997,"15min":328.983},"last":8.9909999999999997},{"interface":"front","average":{"1min":25.178000000000001,"5min":26.393999999999998,"15min":30.449000000000002},"min":{"1min":11.391,"5min":11.391,"15min":10.986000000000001},"max":{"1min":124.36799999999999,"5min":133.02199999999999,"15min":306.81799999999998},"last":17.527000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":17.791,"5min":16.997,"15min":22.556000000000001},"min":{"1min":7.8630000000000004,"5min":7.2549999999999999,"15min":6.9669999999999996},"max":{"1min":127.804,"5min":127.804,"15min":307.69499999999999},"last":9.9990000000000006},{"interface":"front","average":{"1min":21.841999999999999,"5min":22.962,"15min":28.602},"min":{"1min":10.927,"5min":8.9550000000000001,"15min":8.5700000000000003},"max":{"1min":129.26599999999999,"5min":129.26599999999999,"15min":317.13400000000001},"last":10.691000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83092,"kb_used_data":7492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288748,"statfs":{"total":96636764160,"available":96551677952,"internally_reserved":0,"allocated":7671808,"data_stored":4240466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77396427},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":18.553000000000001,"5min":17.256,"15min":18.238},"min":{"1min":4.6230000000000002,"5min":4.6230000000000002,"15min":4.1580000000000004},"max":{"1min":172.93199999999999,"5min":179.608,"15min":259.12200000000001},"last":5.548},{"interface":"front","average":{"1min":19.984999999999999,"5min":18.443999999999999,"15min":18.128},"min":{"1min":4.5999999999999996,"5min":4.5999999999999996,"15min":4.3959999999999999},"max":{"1min":176.59399999999999,"5min":176.59399999999999,"15min":258.25799999999998},"last":7.1600000000000001}]},{"osd":1,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":13.958,"5min":15.481,"15min":18.420999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.2939999999999996},"max":{"1min":50.183,"5min":83.355000000000004,"15min":249.69900000000001},"last":16.523},{"interface":"front","average":{"1min":16.277000000000001,"5min":19.504000000000001,"15min":24.225999999999999},"min":{"1min":8.0890000000000004,"5min":8.0640000000000001,"15min":5.0439999999999996},"max":{"1min":67.171000000000006,"5min":199.15299999999999,"15min":334.67599999999999},"last":15.115}]},{"osd":2,"last update":"Mon Oct 30 17:54:03 2023","interfaces":[{"interface":"back","average":{"1min":18.440999999999999,"5min":18.277999999999999,"15min":22.067},"min":{"1min":7.282,"5min":6.3410000000000002,"15min":6.3140000000000001},"max":{"1min":107.964,"5min":128.202,"15min":403.90899999999999},"last":17.457000000000001},{"interface":"front","average":{"1min":33.597000000000001,"5min":25.751999999999999,"15min":27.367999999999999},"min":{"1min":9.8949999999999996,"5min":8.4559999999999995,"15min":7.6829999999999998},"max":{"1min":276.67700000000002,"5min":276.67700000000002,"15min":354.34800000000001},"last":25.731000000000002}]},{"osd":3,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":19.672000000000001,"5min":165.75800000000001,"15min":72.480000000000004},"min":{"1min":8.8859999999999992,"5min":8.8859999999999992,"15min":6.226},"max":{"1min":71.772999999999996,"5min":8740.9120000000003,"15min":8740.9120000000003},"last":23.808},{"interface":"front","average":{"1min":16.125,"5min":162.107,"15min":70.201999999999998},"min":{"1min":7.444,"5min":7.3840000000000003,"15min":6.1120000000000001},"max":{"1min":50.947000000000003,"5min":8827.2800000000007,"15min":8827.2800000000007},"last":21.263000000000002}]},{"osd":4,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":17.946999999999999,"5min":23.007999999999999,"15min":24.891999999999999},"min":{"1min":6.758,"5min":6.758,"15min":6.758},"max":{"1min":73.082999999999998,"5min":193.517,"15min":265.78300000000002},"last":20.126999999999999},{"interface":"front","average":{"1min":25.338000000000001,"5min":28.114999999999998,"15min":26.849},"min":{"1min":11.759,"5min":10.228999999999999,"15min":8.5839999999999996},"max":{"1min":121.331,"5min":204.27799999999999,"15min":297.99900000000002},"last":26.582000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:54:03 2023","interfaces":[{"interface":"back","average":{"1min":21.972000000000001,"5min":25.536000000000001,"15min":30.370000000000001},"min":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":7.7729999999999997},"max":{"1min":96.088999999999999,"5min":235.08500000000001,"15min":396.77300000000002},"last":15.843999999999999},{"interface":"front","average":{"1min":20.291,"5min":21.542000000000002,"15min":25.526},"min":{"1min":8.0500000000000007,"5min":6.5010000000000003,"15min":5.5010000000000003},"max":{"1min":103.301,"5min":266.30200000000002,"15min":333.88200000000001},"last":21.800999999999998}]},{"osd":6,"last update":"Mon Oct 30 17:54:03 2023","interfaces":[{"interface":"back","average":{"1min":22.289000000000001,"5min":23.263000000000002,"15min":25.934999999999999},"min":{"1min":9.2609999999999992,"5min":9.1809999999999992,"15min":6.9400000000000004},"max":{"1min":68.593999999999994,"5min":174.517,"15min":445.44799999999998},"last":23.141999999999999},{"interface":"front","average":{"1min":31.324999999999999,"5min":28.215,"15min":29.988},"min":{"1min":13.523,"5min":10.054,"15min":8.0060000000000002},"max":{"1min":100.619,"5min":205.42500000000001,"15min":400.87599999999998},"last":22.446999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542418,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78916,"kb_used_data":7476,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292924,"statfs":{"total":96636764160,"available":96555954176,"internally_reserved":0,"allocated":7655424,"data_stored":4210307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":9.3670000000000009,"5min":16.529,"15min":18.324000000000002},"min":{"1min":4.7869999999999999,"5min":4.3339999999999996,"15min":4.1840000000000002},"max":{"1min":26.079000000000001,"5min":135.98099999999999,"15min":254.899},"last":12.679},{"interface":"front","average":{"1min":11.316000000000001,"5min":17.039999999999999,"15min":19.542000000000002},"min":{"1min":6.4569999999999999,"5min":5.4219999999999997,"15min":5.4219999999999997},"max":{"1min":28.175999999999998,"5min":97.100999999999999,"15min":270.51600000000002},"last":16.597000000000001}]},{"osd":1,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":19.887,"5min":20.942,"15min":20.899999999999999},"min":{"1min":7.5099999999999998,"5min":7.2350000000000003,"15min":5.1120000000000001},"max":{"1min":138.97300000000001,"5min":149.53899999999999,"15min":177.15799999999999},"last":7.3620000000000001},{"interface":"front","average":{"1min":12.612,"5min":20.626000000000001,"15min":23.800999999999998},"min":{"1min":7.5529999999999999,"5min":6.2199999999999998,"15min":5.883},"max":{"1min":28.498000000000001,"5min":221.483,"15min":309.04199999999997},"last":13.871}]},{"osd":2,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":14.561999999999999,"5min":19.271000000000001,"15min":20.66},"min":{"1min":7.1379999999999999,"5min":6.8929999999999998,"15min":5.859},"max":{"1min":31.204000000000001,"5min":122.018,"15min":231.15600000000001},"last":14.536},{"interface":"front","average":{"1min":16.201000000000001,"5min":24.071999999999999,"15min":23.096},"min":{"1min":11.792,"5min":11.173999999999999,"15min":6.9889999999999999},"max":{"1min":26.43,"5min":149.17500000000001,"15min":197.58699999999999},"last":15.353}]},{"osd":3,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":9.8900000000000006,"5min":253.59399999999999,"15min":99.061000000000007},"min":{"1min":6.6689999999999996,"5min":5.5039999999999996,"15min":5.3849999999999998},"max":{"1min":14.733000000000001,"5min":7958.2420000000002,"15min":7958.2420000000002},"last":5.9400000000000004},{"interface":"front","average":{"1min":13.827,"5min":254.506,"15min":99.292000000000002},"min":{"1min":8.7859999999999996,"5min":6.7009999999999996,"15min":5.9859999999999998},"max":{"1min":23.07,"5min":7971.3900000000003,"15min":7971.3900000000003},"last":8.3620000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":15.43,"5min":25.451000000000001,"15min":25.949000000000002},"min":{"1min":10.375999999999999,"5min":7.5300000000000002,"15min":7.5300000000000002},"max":{"1min":22.405000000000001,"5min":236.994,"15min":262.15699999999998},"last":17.315999999999999},{"interface":"front","average":{"1min":16.157,"5min":23.646999999999998,"15min":23.992000000000001},"min":{"1min":9.6839999999999993,"5min":8.6479999999999997,"15min":6.7859999999999996},"max":{"1min":23.186,"5min":228.88200000000001,"15min":228.88200000000001},"last":10.827999999999999}]},{"osd":5,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":11.866,"5min":20.151,"15min":24.649999999999999},"min":{"1min":6.5540000000000003,"5min":6.2359999999999998,"15min":5.7809999999999997},"max":{"1min":18.652000000000001,"5min":191.154,"15min":289.22699999999998},"last":8.9949999999999992},{"interface":"front","average":{"1min":17.635000000000002,"5min":25.885999999999999,"15min":27.363},"min":{"1min":14.632999999999999,"5min":10.679,"15min":7.835},"max":{"1min":25.047000000000001,"5min":230.613,"15min":230.613},"last":17.914999999999999}]},{"osd":7,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":27.414000000000001,"5min":24.706,"15min":26.167999999999999},"min":{"1min":9.2230000000000008,"5min":8.0370000000000008,"15min":7.2729999999999997},"max":{"1min":164.86600000000001,"5min":192.72399999999999,"15min":290.22300000000001},"last":10.035},{"interface":"front","average":{"1min":27.791,"5min":26.841000000000001,"15min":28.561},"min":{"1min":12.119999999999999,"5min":9.5440000000000005,"15min":7.4489999999999998},"max":{"1min":130.845,"5min":270.01900000000001,"15min":270.01900000000001},"last":15.973000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542418,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60948,"kb_used_data":7552,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310892,"statfs":{"total":96636764160,"available":96574353408,"internally_reserved":0,"allocated":7733248,"data_stored":4297155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":8.5139999999999993,"5min":21.638000000000002,"15min":19.297000000000001},"min":{"1min":4.5659999999999998,"5min":3.7290000000000001,"15min":3.7290000000000001},"max":{"1min":31.369,"5min":278.70299999999997,"15min":317.529},"last":8.1470000000000002},{"interface":"front","average":{"1min":8.5500000000000007,"5min":19.698,"15min":19.140999999999998},"min":{"1min":4.9850000000000003,"5min":4.9850000000000003,"15min":4.8040000000000003},"max":{"1min":32.368000000000002,"5min":174.08099999999999,"15min":200.29300000000001},"last":9.1690000000000005}]},{"osd":1,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":14.672000000000001,"5min":18,"15min":23.431999999999999},"min":{"1min":5.6029999999999998,"5min":5.6029999999999998,"15min":5.6029999999999998},"max":{"1min":25.289999999999999,"5min":136.81200000000001,"15min":307.86799999999999},"last":20.815999999999999},{"interface":"front","average":{"1min":12.586,"5min":18.454999999999998,"15min":23.405000000000001},"min":{"1min":7.9080000000000004,"5min":6.1669999999999998,"15min":6.0819999999999999},"max":{"1min":30.154,"5min":139.191,"15min":307.05500000000001},"last":6.8570000000000002}]},{"osd":2,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":14.393000000000001,"5min":19.882999999999999,"15min":24.52},"min":{"1min":7.2400000000000002,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":25.452999999999999,"5min":279.37099999999998,"15min":283.87599999999998},"last":112.958},{"interface":"front","average":{"1min":24.056000000000001,"5min":21.108000000000001,"15min":22.945},"min":{"1min":9.3710000000000004,"5min":6.9390000000000001,"15min":5.8259999999999996},"max":{"1min":198.02500000000001,"5min":198.02500000000001,"15min":321.54000000000002},"last":92.144999999999996}]},{"osd":3,"last update":"Mon Oct 30 17:54:12 2023","interfaces":[{"interface":"back","average":{"1min":27.785,"5min":165.78,"15min":72.799000000000007},"min":{"1min":9.952,"5min":5.9219999999999997,"15min":5.9219999999999997},"max":{"1min":247.03399999999999,"5min":8219.1820000000007,"15min":8219.1820000000007},"last":22.355},{"interface":"front","average":{"1min":14.07,"5min":161.76900000000001,"15min":71.828000000000003},"min":{"1min":8.7919999999999998,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":34.832999999999998,"5min":8140.4719999999998,"15min":8140.4719999999998},"last":25.198}]},{"osd":5,"last update":"Mon Oct 30 17:54:53 2023","interfaces":[{"interface":"back","average":{"1min":18.893000000000001,"5min":22.899000000000001,"15min":23.814},"min":{"1min":9.1790000000000003,"5min":6.4169999999999998,"15min":6.4169999999999998},"max":{"1min":65.632000000000005,"5min":292.22800000000001,"15min":292.22800000000001},"last":11.455},{"interface":"front","average":{"1min":18.199999999999999,"5min":22.989999999999998,"15min":22.789999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":59.558,"5min":179.67500000000001,"15min":180.28800000000001},"last":12.792999999999999}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":28.52,"5min":26.338999999999999,"15min":25.452000000000002},"min":{"1min":12.355,"5min":9.7889999999999997,"15min":6.5810000000000004},"max":{"1min":191.995,"5min":191.995,"15min":293.16300000000001},"last":12.082000000000001},{"interface":"front","average":{"1min":17.751999999999999,"5min":26.558,"15min":27.359999999999999},"min":{"1min":8.516,"5min":8.2420000000000009,"15min":7.6779999999999999},"max":{"1min":43.220999999999997,"5min":207.30099999999999,"15min":281.62799999999999},"last":23.100000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":18.670000000000002,"5min":26.806999999999999,"15min":28.308},"min":{"1min":12.002000000000001,"5min":8.0289999999999999,"15min":7.4969999999999999},"max":{"1min":43.93,"5min":396.928,"15min":396.928},"last":9.9209999999999994},{"interface":"front","average":{"1min":18.928000000000001,"5min":32.037999999999997,"15min":29.954999999999998},"min":{"1min":10.805999999999999,"5min":8.9809999999999999,"15min":7.0860000000000003},"max":{"1min":46.781999999999996,"5min":366.32499999999999,"15min":366.32499999999999},"last":21.715}]}]},{"osd":5,"up_from":13,"seq":55834575122,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79328,"kb_used_data":7888,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292512,"statfs":{"total":96636764160,"available":96555532288,"internally_reserved":0,"allocated":8077312,"data_stored":4637651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.609999999999999,"5min":14.541,"15min":16.334},"min":{"1min":4.7089999999999996,"5min":3.9910000000000001,"15min":3.9569999999999999},"max":{"1min":83.933999999999997,"5min":109.919,"15min":155.58799999999999},"last":6.1840000000000002},{"interface":"front","average":{"1min":17.244,"5min":16.513999999999999,"15min":18.204999999999998},"min":{"1min":7.4370000000000003,"5min":4.8789999999999996,"15min":4.6900000000000004},"max":{"1min":82.497,"5min":159.01900000000001,"15min":180.94499999999999},"last":15.596}]},{"osd":1,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":10.683999999999999,"5min":18.123999999999999,"15min":20.596},"min":{"1min":5.0970000000000004,"5min":5.0970000000000004,"15min":4.8810000000000002},"max":{"1min":30.02,"5min":228.15199999999999,"15min":233.22800000000001},"last":16.308},{"interface":"front","average":{"1min":14.571,"5min":15.516999999999999,"15min":18.213000000000001},"min":{"1min":8.5969999999999995,"5min":5.8259999999999996,"15min":5.1120000000000001},"max":{"1min":28.818999999999999,"5min":128.197,"15min":278.41300000000001},"last":18.452000000000002}]},{"osd":2,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":16.053999999999998,"5min":16.785,"15min":19.841000000000001},"min":{"1min":7.101,"5min":7.101,"15min":5.8949999999999996},"max":{"1min":30.600999999999999,"5min":135.75999999999999,"15min":234.09299999999999},"last":17.797000000000001},{"interface":"front","average":{"1min":13.555999999999999,"5min":15.548,"15min":20.547000000000001},"min":{"1min":6.657,"5min":6.5510000000000002,"15min":6.5060000000000002},"max":{"1min":27.719999999999999,"5min":87.316999999999993,"15min":277.52300000000002},"last":19.933}]},{"osd":3,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":13.231999999999999,"5min":151.136,"15min":65.766999999999996},"min":{"1min":6.2080000000000002,"5min":6.2080000000000002,"15min":6.2080000000000002},"max":{"1min":48.613999999999997,"5min":6785.9989999999998,"15min":6785.9989999999998},"last":16.978000000000002},{"interface":"front","average":{"1min":16.672999999999998,"5min":151.255,"15min":65.844999999999999},"min":{"1min":6.6870000000000003,"5min":6.6870000000000003,"15min":6.2839999999999998},"max":{"1min":54.656999999999996,"5min":6786.7420000000002,"15min":6786.7420000000002},"last":31.672000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.928999999999998,"15min":22.541},"min":{"1min":7.2590000000000003,"5min":6.9790000000000001,"15min":6.6609999999999996},"max":{"1min":24.433,"5min":143.75800000000001,"15min":286.96699999999998},"last":19.233000000000001},{"interface":"front","average":{"1min":18.093,"5min":23.826000000000001,"15min":26.565000000000001},"min":{"1min":12.166,"5min":12.166,"15min":8.609},"max":{"1min":26.777000000000001,"5min":144.934,"15min":276.17200000000003},"last":21.702000000000002}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.897,"5min":17.413,"15min":21.896000000000001},"min":{"1min":7.2969999999999997,"5min":6.5190000000000001,"15min":6.5190000000000001},"max":{"1min":77.432000000000002,"5min":77.432000000000002,"15min":280.36000000000001},"last":30.943999999999999},{"interface":"front","average":{"1min":16.088000000000001,"5min":21.170999999999999,"15min":26.117000000000001},"min":{"1min":9.4410000000000007,"5min":7.4020000000000001,"15min":7.4020000000000001},"max":{"1min":25.859999999999999,"5min":109.157,"15min":346.47000000000003},"last":32.313000000000002}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":17.742999999999999,"5min":21.178000000000001,"15min":28.771999999999998},"min":{"1min":8.8689999999999998,"5min":8.8689999999999998,"15min":7.6520000000000001},"max":{"1min":54.5,"5min":110.94799999999999,"15min":332.99099999999999},"last":170.19399999999999},{"interface":"front","average":{"1min":20.643000000000001,"5min":22.628,"15min":29.914999999999999},"min":{"1min":11.954000000000001,"5min":10.728,"15min":9.6750000000000007},"max":{"1min":72.408000000000001,"5min":145.00899999999999,"15min":332.64800000000002},"last":38.954000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T17:55:01.455 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T17:55:01.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.555+0000 7f3f81f97640 10 monclient: get_monmap_and_config 2023-10-30T17:55:01.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.555+0000 7f3f81f97640 10 monclient: build_initial_monmap 2023-10-30T17:55:01.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.555+0000 7f3f81f97640 10 monclient: monmap: 2023-10-30T17:55:01.557 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:01.557 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:01.557 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:01.556112+0000 2023-10-30T17:55:01.558 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:01.556112+0000 2023-10-30T17:55:01.558 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:01.558 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.558 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:01.558 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:01.559 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:01.559 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:01.559 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:01.559 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.562 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.555+0000 7f3f81f97640 1 Processor -- start 2023-10-30T17:55:01.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 1 -- start start 2023-10-30T17:55:01.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 10 monclient: init 2023-10-30T17:55:01.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:01.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-10-30T17:55:01.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f7c0a84b0 0x7f3f7c13f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:01.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 10 monclient(hunting): picked mon.noname-b con 0x7f3f7c0a84b0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:01.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3f7c0a8890 0x7f3f7c141d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:01.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c0a8890 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:01.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f7c0a9250 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:01.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c142960 con 0x7f3f7c0a9250 2023-10-30T17:55:01.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:01.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c0a9670 con 0x7f3f7c0a8890 2023-10-30T17:55:01.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:01.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c0aa9d0 con 0x7f3f7c0a84b0 2023-10-30T17:55:01.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:01.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.556+0000 7f3f81f97640 10 monclient(hunting): authenticate will time out at 2683.879817s 2023-10-30T17:55:01.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f7b7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f7c0a84b0 0x7f3f7c13f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:01.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f7bfff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f3f7c0a9250 0x7f3f7c142250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59404/0 (socket says 172.21.15.170:59404) 2023-10-30T17:55:01.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f7bfff640 1 -- 172.21.15.170:0/2189270071 learned_addr learned my addr 172.21.15.170:0/2189270071 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:01.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f72ffd640 1 --2- 172.21.15.170:0/2189270071 >> v2:172.21.15.177:3300/0 conn(0x7f3f7c0a8890 0x7f3f7c141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:01.567 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f72ffd640 10 monclient(hunting): get_auth_request con 0x7f3f7c0a8890 auth_method 0 2023-10-30T17:55:01.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:01.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f72ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:01.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:01.568 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f7b7fe640 10 monclient(hunting): get_auth_request con 0x7f3f7c0a84b0 auth_method 0 2023-10-30T17:55:01.569 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:01.569 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:01.569 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.557+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:01.569 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.558+0000 7f3f72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:01.570 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.558+0000 7f3f72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:01.570 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.558+0000 7f3f72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:01.570 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.558+0000 7f3f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:01.570 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.558+0000 7f3f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:01.570 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.558+0000 7f3f7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:01.571 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.559+0000 7f3f72ffd640 10 monclient(hunting): handle_auth_done global_id 21427 payload 274 2023-10-30T17:55:01.571 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.559+0000 7f3f72ffd640 1 -- 172.21.15.170:0/2189270071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f7c0a84b0 msgr2=0x7f3f7c13f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:01.571 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.559+0000 7f3f72ffd640 1 --2- 172.21.15.170:0/2189270071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f7c0a84b0 0x7f3f7c13f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.571 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.559+0000 7f3f72ffd640 1 -- 172.21.15.170:0/2189270071 >> v1:172.21.15.170:6789/0 conn(0x7f3f7c0a9250 legacy=0x7f3f7c142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:01.571 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.559+0000 7f3f72ffd640 10 monclient: _finish_hunting 0 2023-10-30T17:55:01.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.559+0000 7f3f72ffd640 1 monclient: found mon.noname-c 2023-10-30T17:55:01.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.559+0000 7f3f72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:01.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.559+0000 7f3f72ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:55:01.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.559+0000 7f3f72ffd640 1 -- 172.21.15.170:0/2189270071 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c0b1570 con 0x7f3f7c0a8890 2023-10-30T17:55:01.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.561+0000 7f3f72ffd640 1 --2- 172.21.15.170:0/2189270071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a8890 0x7f3f7c141d10 secure :-1 s=READY pgs=5264 cs=0 l=1 rev1=1 crypto rx=0x7f3f640097d0 tx=0x7f3f6402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f18930e105d5c6e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:01.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 1 -- 172.21.15.170:0/2189270071 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f640316f0 con 0x7f3f7c0a8890 2023-10-30T17:55:01.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:01.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:01.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 10 monclient: dump: 2023-10-30T17:55:01.578 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:01.578 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:01.579 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.579 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.579 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:01.579 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.580 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:01.580 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:01.580 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:01.580 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 10 monclient: _finish_auth 0 2023-10-30T17:55:01.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 10 monclient: _check_auth_tickets 2023-10-30T17:55:01.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:01.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 1 -- 172.21.15.170:0/2189270071 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f64031850 con 0x7f3f7c0a8890 2023-10-30T17:55:01.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 5 monclient: authenticate success, global_id 21427 2023-10-30T17:55:01.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:01.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:01.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:01.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 4 set_mon_vals no callback set 2023-10-30T17:55:01.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 1 -- 172.21.15.170:0/2189270071 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64031be0 con 0x7f3f7c0a8890 2023-10-30T17:55:01.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:01.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:01.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f7affd640 10 monclient: dump: 2023-10-30T17:55:01.583 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:01.583 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:01.584 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.584 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.584 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:01.584 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.584 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:01.585 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:01.585 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:01.585 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 10 monclient: shutdown 2023-10-30T17:55:01.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:01.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 1 -- 172.21.15.170:0/2189270071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a8890 msgr2=0x7f3f7c141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:01.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 1 --2- 172.21.15.170:0/2189270071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a8890 0x7f3f7c141d10 secure :-1 s=READY pgs=5264 cs=0 l=1 rev1=1 crypto rx=0x7f3f640097d0 tx=0x7f3f6402b790 comp rx=0 tx=0).stop 2023-10-30T17:55:01.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 1 -- 172.21.15.170:0/2189270071 shutdown_connections 2023-10-30T17:55:01.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 1 --2- 172.21.15.170:0/2189270071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a8890 0x7f3f7c141d10 unknown :-1 s=CLOSED pgs=5264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.587 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 1 --2- 172.21.15.170:0/2189270071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f7c0a84b0 0x7f3f7c13f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.587 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 1 -- 172.21.15.170:0/2189270071 >> 172.21.15.170:0/2189270071 conn(0x7f3f7c0b7330 msgr2=0x7f3f7c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:01.587 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 1 -- 172.21.15.170:0/2189270071 shutdown_connections 2023-10-30T17:55:01.587 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.565+0000 7f3f81f97640 1 -- 172.21.15.170:0/2189270071 wait complete. 2023-10-30T17:55:01.587 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.566+0000 7f3f81f97640 10 monclient: build_initial_monmap 2023-10-30T17:55:01.588 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.566+0000 7f3f81f97640 10 monclient: monmap: 2023-10-30T17:55:01.588 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:01.588 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:01.588 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:01.588 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:01.589 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:01.589 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.589 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:01.589 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:01.589 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:01.590 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.590 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.566+0000 7f3f81f97640 1 librados: starting msgr at 2023-10-30T17:55:01.590 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.566+0000 7f3f81f97640 1 librados: starting objecter 2023-10-30T17:55:01.590 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.566+0000 7f3f81f97640 1 Processor -- start 2023-10-30T17:55:01.591 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 1 -- start start 2023-10-30T17:55:01.591 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 1 librados: setting wanted keys 2023-10-30T17:55:01.591 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 1 librados: calling monclient init 2023-10-30T17:55:01.591 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient: init 2023-10-30T17:55:01.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:01.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:55:01.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a84b0 0x7f3f7c0abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:01.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient(hunting): picked mon.noname-b con 0x7f3f7c0a84b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:01.592 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f7c0a9250 0x7f3f7c13b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:01.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c0a9250 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:01.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f7c0adc90 0x7f3f7c13b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:01.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c0adc90 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:01.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:01.593 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c0ac4d0 con 0x7f3f7c0a9250 2023-10-30T17:55:01.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:01.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c13bbb0 con 0x7f3f7c0a84b0 2023-10-30T17:55:01.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:01.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c0ac180 con 0x7f3f7c0adc90 2023-10-30T17:55:01.594 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:01.595 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.567+0000 7f3f81f97640 10 monclient(hunting): authenticate will time out at 2683.890755s 2023-10-30T17:55:01.595 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f7c0adc90 0x7f3f7c13b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:01.595 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f72ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f7c0a9250 0x7f3f7c13b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:01.595 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a84b0 0x7f3f7c0abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:01.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a84b0 0x7f3f7c0abc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:49442/0 (socket says 172.21.15.170:49442) 2023-10-30T17:55:01.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7b7fe640 1 -- 172.21.15.170:0/3991501291 learned_addr learned my addr 172.21.15.170:0/3991501291 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:01.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7b7fe640 10 monclient(hunting): get_auth_request con 0x7f3f7c0a84b0 auth_method 0 2023-10-30T17:55:01.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:01.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:01.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:01.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f72ffd640 10 monclient(hunting): get_auth_request con 0x7f3f7c0a9250 auth_method 0 2023-10-30T17:55:01.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:01.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f72ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:01.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:01.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7bfff640 10 monclient(hunting): get_auth_request con 0x7f3f7c0adc90 auth_method 0 2023-10-30T17:55:01.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:01.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:01.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:01.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:01.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:01.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:01.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.568+0000 7f3f72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:01.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.569+0000 7f3f72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:01.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.569+0000 7f3f72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:01.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.569+0000 7f3f7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:01.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.569+0000 7f3f7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:01.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.569+0000 7f3f7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:01.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f72ffd640 10 monclient(hunting): handle_auth_done global_id 15981 payload 824 2023-10-30T17:55:01.601 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f72ffd640 1 -- 172.21.15.170:0/3991501291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f7c0adc90 msgr2=0x7f3f7c13b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:01.601 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f72ffd640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f7c0adc90 0x7f3f7c13b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.601 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f72ffd640 1 -- 172.21.15.170:0/3991501291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a84b0 msgr2=0x7f3f7c0abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:01.601 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f72ffd640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a84b0 0x7f3f7c0abc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:01.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f72ffd640 10 monclient: _finish_hunting 0 2023-10-30T17:55:01.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f72ffd640 1 monclient: found mon.noname-a 2023-10-30T17:55:01.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:01.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f72ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:01.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f72ffd640 1 -- 172.21.15.170:0/3991501291 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f64009400 con 0x7f3f7c0a9250 2023-10-30T17:55:01.603 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.571+0000 7f3f7b7fe640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a84b0 0x7f3f7c0abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T17:55:01.603 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.574+0000 7f3f72ffd640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f7c0a9250 0x7f3f7c13b130 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f3f64033000 tx=0x7f3f640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:01.603 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.574+0000 7f3f78ff9640 10 client.?.objecter ms_handle_connect 0x7f3f7c0a9250 2023-10-30T17:55:01.603 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.574+0000 7f3f78ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:01.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 1 -- 172.21.15.170:0/3991501291 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64034680 con 0x7f3f7c0a9250 2023-10-30T17:55:01.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:01.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:01.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 10 monclient: dump: 2023-10-30T17:55:01.606 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:01.607 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:01.607 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.607 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.607 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:01.607 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.608 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:01.608 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:01.608 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:01.608 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 10 monclient: _finish_auth 0 2023-10-30T17:55:01.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 10 monclient: _check_auth_tickets 2023-10-30T17:55:01.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:01.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 1 -- 172.21.15.170:0/3991501291 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f64034800 con 0x7f3f7c0a9250 2023-10-30T17:55:01.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:01.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 1 -- 172.21.15.170:0/3991501291 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64034bb0 con 0x7f3f7c0a9250 2023-10-30T17:55:01.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:01.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:01.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f78ff9640 10 monclient: dump: 2023-10-30T17:55:01.610 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:01.611 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:01.611 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.611 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:01.611 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:01.612 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:01.612 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:01.612 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:01.612 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:01.612 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:01.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f81f97640 5 monclient: authenticate success, global_id 15981 2023-10-30T17:55:01.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f81f97640 10 monclient: _renew_subs 2023-10-30T17:55:01.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f81f97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:01.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f81f97640 1 -- 172.21.15.170:0/3991501291 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c145ab0 con 0x7f3f7c0a9250 2023-10-30T17:55:01.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f7a7fc640 4 set_mon_vals no callback set 2023-10-30T17:55:01.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f81f97640 10 client.15981.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:01.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f81f97640 10 monclient: _renew_subs 2023-10-30T17:55:01.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f81f97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:01.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f81f97640 1 -- 172.21.15.170:0/3991501291 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c13bfc0 con 0x7f3f7c0a9250 2023-10-30T17:55:01.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.577+0000 7f3f81f97640 1 librados: init done 2023-10-30T17:55:01.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.578+0000 7f3f81f97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:01.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.578+0000 7f3f81f97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:01.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.578+0000 7f3f81f97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:01.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.578+0000 7f3f81f97640 1 -- 172.21.15.170:0/3991501291 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7c0ac640 con 0x7f3f7c0a9250 2023-10-30T17:55:01.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.596+0000 7f3f78ff9640 1 -- 172.21.15.170:0/3991501291 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3f6403b040 con 0x7f3f7c0a9250 2023-10-30T17:55:01.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.596+0000 7f3f78ff9640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3f5c06c850 0x7f3f5c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:01.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.597+0000 7f3f7b7fe640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3f5c06c850 0x7f3f5c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:01.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.597+0000 7f3f7b7fe640 10 monclient: get_auth_request con 0x7f3f5c06c850 auth_method 0 2023-10-30T17:55:01.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.597+0000 7f3f78ff9640 10 client.15981.objecter ms_handle_connect 0x7f3f5c06c850 2023-10-30T17:55:01.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.597+0000 7f3f7b7fe640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3f5c06c850 0x7f3f5c06ed10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0112c0 tx=0x7f3f6c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:01.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.599+0000 7f3f78ff9640 1 -- 172.21.15.170:0/3991501291 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(913..913 src has 374..913) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f3f64079ee0 con 0x7f3f7c0a9250 2023-10-30T17:55:01.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.599+0000 7f3f78ff9640 10 client.15981.objecter ms_dispatch 0x7f3f7c000b90 osd_map(913..913 src has 374..913) v4 2023-10-30T17:55:01.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.599+0000 7f3f78ff9640 3 client.15981.objecter handle_osd_map got epochs [913,913] > 0 2023-10-30T17:55:01.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.599+0000 7f3f78ff9640 3 client.15981.objecter handle_osd_map decoding full epoch 913 2023-10-30T17:55:01.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.599+0000 7f3f78ff9640 20 client.15981.objecter dump_active .. 0 homeless 2023-10-30T17:55:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.830+0000 7f3f78ff9640 1 -- 172.21.15.170:0/3991501291 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3f64076220 con 0x7f3f7c0a9250 2023-10-30T17:55:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.830+0000 7f3f78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.830+0000 7f3f78ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:01.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.992+0000 7f3f81f97640 1 -- 172.21.15.170:0/3991501291 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3f7c0ac2c0 con 0x7f3f5c06c850 2023-10-30T17:55:01.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.997+0000 7f3f78ff9640 1 -- 172.21.15.170:0/3991501291 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151540 (secure 0 0 0) 0x7f3f6c009d20 con 0x7f3f5c06c850 2023-10-30T17:55:01.999 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:55:02.000 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-10-30T17:55:02.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 10 librados: watch_flush enter 2023-10-30T17:55:02.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 10 librados: watch_flush exit 2023-10-30T17:55:02.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 20 client.15981.objecter shutdown clearing up homeless session... 2023-10-30T17:55:02.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 10 client.15981.objecter successfully canceled tick 2023-10-30T17:55:02.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 -- 172.21.15.170:0/3991501291 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3f5c06c850 msgr2=0x7f3f5c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3f5c06c850 0x7f3f5c06ed10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0112c0 tx=0x7f3f6c009190 comp rx=0 tx=0).stop 2023-10-30T17:55:02.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 10 monclient: shutdown 2023-10-30T17:55:02.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:02.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 -- 172.21.15.170:0/3991501291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f7c0a9250 msgr2=0x7f3f7c13b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f7c0a9250 0x7f3f7c13b130 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f3f64033000 tx=0x7f3f640046c0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 -- 172.21.15.170:0/3991501291 shutdown_connections 2023-10-30T17:55:02.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f3f5c06c850 0x7f3f5c06ed10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f7c0adc90 0x7f3f7c13b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f7c0a9250 0x7f3f7c13b130 unknown :-1 s=CLOSED pgs=3361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 --2- 172.21.15.170:0/3991501291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f7c0a84b0 0x7f3f7c0abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 -- 172.21.15.170:0/3991501291 >> 172.21.15.170:0/3991501291 conn(0x7f3f7c0b7330 msgr2=0x7f3f7c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:02.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 -- 172.21.15.170:0/3991501291 shutdown_connections 2023-10-30T17:55:02.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 -- 172.21.15.170:0/3991501291 wait complete. 2023-10-30T17:55:02.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:01.999+0000 7f3f81f97640 1 librados: shutdown 2023-10-30T17:55:02.012 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2023-10-30T17:55:01.325774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":568424,"kb_used_data":60776,"kb_used_omap":12,"kb_used_meta":507507,"kb_avail":754406296,"statfs":{"total":773094113280,"available":772512047104,"internally_reserved":0,"allocated":62234624,"data_stored":34698846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":519687736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":375,"apply_latency_ms":375,"commit_latency_ns":375000000,"apply_latency_ns":375000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000948"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":856,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.965106+0000","last_change":"2023-10-30T17:33:37.648571+0000","last_active":"2023-10-30T17:54:56.965106+0000","last_peered":"2023-10-30T17:54:56.965106+0000","last_clean":"2023-10-30T17:54:56.965106+0000","last_became_active":"2023-10-30T17:33:37.643213+0000","last_became_peered":"2023-10-30T17:33:37.643213+0000","last_unstale":"2023-10-30T17:54:56.965106+0000","last_undegraded":"2023-10-30T17:54:56.965106+0000","last_fullsized":"2023-10-30T17:54:56.965106+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:43:30.718487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":853,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.252428+0000","last_change":"2023-10-30T17:33:37.836041+0000","last_active":"2023-10-30T17:54:59.252428+0000","last_peered":"2023-10-30T17:54:59.252428+0000","last_clean":"2023-10-30T17:54:59.252428+0000","last_became_active":"2023-10-30T17:33:37.827806+0000","last_became_peered":"2023-10-30T17:33:37.827806+0000","last_unstale":"2023-10-30T17:54:59.252428+0000","last_undegraded":"2023-10-30T17:54:59.252428+0000","last_fullsized":"2023-10-30T17:54:59.252428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:00.255976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":851,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.737796+0000","last_change":"2023-10-30T17:33:40.825119+0000","last_active":"2023-10-30T17:54:55.737796+0000","last_peered":"2023-10-30T17:54:55.737796+0000","last_clean":"2023-10-30T17:54:55.737796+0000","last_became_active":"2023-10-30T17:33:40.511287+0000","last_became_peered":"2023-10-30T17:33:40.511287+0000","last_unstale":"2023-10-30T17:54:55.737796+0000","last_undegraded":"2023-10-30T17:54:55.737796+0000","last_fullsized":"2023-10-30T17:54:55.737796+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:14:23.464432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":853,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.248681+0000","last_change":"2023-10-30T17:33:38.590694+0000","last_active":"2023-10-30T17:54:59.248681+0000","last_peered":"2023-10-30T17:54:59.248681+0000","last_clean":"2023-10-30T17:54:59.248681+0000","last_became_active":"2023-10-30T17:33:38.581457+0000","last_became_peered":"2023-10-30T17:33:38.581457+0000","last_unstale":"2023-10-30T17:54:59.248681+0000","last_undegraded":"2023-10-30T17:54:59.248681+0000","last_fullsized":"2023-10-30T17:54:59.248681+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:17.385325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":856,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.939866+0000","last_change":"2023-10-30T17:33:46.635360+0000","last_active":"2023-10-30T17:54:56.939866+0000","last_peered":"2023-10-30T17:54:56.939866+0000","last_clean":"2023-10-30T17:54:56.939866+0000","last_became_active":"2023-10-30T17:33:46.629442+0000","last_became_peered":"2023-10-30T17:33:46.629442+0000","last_unstale":"2023-10-30T17:54:56.939866+0000","last_undegraded":"2023-10-30T17:54:56.939866+0000","last_fullsized":"2023-10-30T17:54:56.939866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:46:06.082348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":852,"reported_epoch":909,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.139730+0000","last_change":"2023-10-30T17:33:39.141659+0000","last_active":"2023-10-30T17:54:57.139730+0000","last_peered":"2023-10-30T17:54:57.139730+0000","last_clean":"2023-10-30T17:54:57.139730+0000","last_became_active":"2023-10-30T17:33:39.135750+0000","last_became_peered":"2023-10-30T17:33:39.135750+0000","last_unstale":"2023-10-30T17:54:57.139730+0000","last_undegraded":"2023-10-30T17:54:57.139730+0000","last_fullsized":"2023-10-30T17:54:57.139730+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:27.698297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":853,"reported_epoch":909,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.207778+0000","last_change":"2023-10-30T17:33:51.011431+0000","last_active":"2023-10-30T17:54:57.207778+0000","last_peered":"2023-10-30T17:54:57.207778+0000","last_clean":"2023-10-30T17:54:57.207778+0000","last_became_active":"2023-10-30T17:33:51.001524+0000","last_became_peered":"2023-10-30T17:33:51.001524+0000","last_unstale":"2023-10-30T17:54:57.207778+0000","last_undegraded":"2023-10-30T17:54:57.207778+0000","last_fullsized":"2023-10-30T17:54:57.207778+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:41.602832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.027235+0000","last_change":"2023-10-30T17:33:58.879816+0000","last_active":"2023-10-30T17:54:58.027235+0000","last_peered":"2023-10-30T17:54:58.027235+0000","last_clean":"2023-10-30T17:54:58.027235+0000","last_became_active":"2023-10-30T17:33:58.874661+0000","last_became_peered":"2023-10-30T17:33:58.874661+0000","last_unstale":"2023-10-30T17:54:58.027235+0000","last_undegraded":"2023-10-30T17:54:58.027235+0000","last_fullsized":"2023-10-30T17:54:58.027235+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:04:24.038125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":849,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.993781+0000","last_change":"2023-10-30T17:33:39.104029+0000","last_active":"2023-10-30T17:54:58.993781+0000","last_peered":"2023-10-30T17:54:58.993781+0000","last_clean":"2023-10-30T17:54:58.993781+0000","last_became_active":"2023-10-30T17:33:39.086390+0000","last_became_peered":"2023-10-30T17:33:39.086390+0000","last_unstale":"2023-10-30T17:54:58.993781+0000","last_undegraded":"2023-10-30T17:54:58.993781+0000","last_fullsized":"2023-10-30T17:54:58.993781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:52.540858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":853,"reported_epoch":909,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.186394+0000","last_change":"2023-10-30T17:33:40.038786+0000","last_active":"2023-10-30T17:54:57.186394+0000","last_peered":"2023-10-30T17:54:57.186394+0000","last_clean":"2023-10-30T17:54:57.186394+0000","last_became_active":"2023-10-30T17:33:39.965244+0000","last_became_peered":"2023-10-30T17:33:39.965244+0000","last_unstale":"2023-10-30T17:54:57.186394+0000","last_undegraded":"2023-10-30T17:54:57.186394+0000","last_fullsized":"2023-10-30T17:54:57.186394+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:17:21.402055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":857,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.954339+0000","last_change":"2023-10-30T17:33:46.567515+0000","last_active":"2023-10-30T17:54:56.954339+0000","last_peered":"2023-10-30T17:54:56.954339+0000","last_clean":"2023-10-30T17:54:56.954339+0000","last_became_active":"2023-10-30T17:33:46.562080+0000","last_became_peered":"2023-10-30T17:33:46.562080+0000","last_unstale":"2023-10-30T17:54:56.954339+0000","last_undegraded":"2023-10-30T17:54:56.954339+0000","last_fullsized":"2023-10-30T17:54:56.954339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:42:52.518433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.979403+0000","last_change":"2023-10-30T17:33:40.831046+0000","last_active":"2023-10-30T17:54:57.979403+0000","last_peered":"2023-10-30T17:54:57.979403+0000","last_clean":"2023-10-30T17:54:57.979403+0000","last_became_active":"2023-10-30T17:33:40.798989+0000","last_became_peered":"2023-10-30T17:33:40.798989+0000","last_unstale":"2023-10-30T17:54:57.979403+0000","last_undegraded":"2023-10-30T17:54:57.979403+0000","last_fullsized":"2023-10-30T17:54:57.979403+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:35.578143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":853,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.023321+0000","last_change":"2023-10-30T17:32:35.372482+0000","last_active":"2023-10-30T17:54:59.023321+0000","last_peered":"2023-10-30T17:54:59.023321+0000","last_clean":"2023-10-30T17:54:59.023321+0000","last_became_active":"2023-10-30T17:32:35.367487+0000","last_became_peered":"2023-10-30T17:32:35.367487+0000","last_unstale":"2023-10-30T17:54:59.023321+0000","last_undegraded":"2023-10-30T17:54:59.023321+0000","last_fullsized":"2023-10-30T17:54:59.023321+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:45:03.846674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":858,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.920758+0000","last_change":"2023-10-30T17:33:38.291772+0000","last_active":"2023-10-30T17:54:56.920758+0000","last_peered":"2023-10-30T17:54:56.920758+0000","last_clean":"2023-10-30T17:54:56.920758+0000","last_became_active":"2023-10-30T17:33:38.284317+0000","last_became_peered":"2023-10-30T17:33:38.284317+0000","last_unstale":"2023-10-30T17:54:56.920758+0000","last_undegraded":"2023-10-30T17:54:56.920758+0000","last_fullsized":"2023-10-30T17:54:56.920758+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:02.583403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":863,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.945069+0000","last_change":"2023-10-30T17:32:34.589393+0000","last_active":"2023-10-30T17:54:56.945069+0000","last_peered":"2023-10-30T17:54:56.945069+0000","last_clean":"2023-10-30T17:54:56.945069+0000","last_became_active":"2023-10-30T17:32:34.557021+0000","last_became_peered":"2023-10-30T17:32:34.557021+0000","last_unstale":"2023-10-30T17:54:56.945069+0000","last_undegraded":"2023-10-30T17:54:56.945069+0000","last_fullsized":"2023-10-30T17:54:56.945069+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:05.352859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"47'7","reported_seq":863,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.992239+0000","last_change":"2023-10-30T17:33:39.222547+0000","last_active":"2023-10-30T17:54:58.992239+0000","last_peered":"2023-10-30T17:54:58.992239+0000","last_clean":"2023-10-30T17:54:58.992239+0000","last_became_active":"2023-10-30T17:33:39.211129+0000","last_became_peered":"2023-10-30T17:33:39.211129+0000","last_unstale":"2023-10-30T17:54:58.992239+0000","last_undegraded":"2023-10-30T17:54:58.992239+0000","last_fullsized":"2023-10-30T17:54:58.992239+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:49.479747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.008700+0000","last_change":"2023-10-30T17:32:35.362652+0000","last_active":"2023-10-30T17:54:59.008700+0000","last_peered":"2023-10-30T17:54:59.008700+0000","last_clean":"2023-10-30T17:54:59.008700+0000","last_became_active":"2023-10-30T17:32:35.357489+0000","last_became_peered":"2023-10-30T17:32:35.357489+0000","last_unstale":"2023-10-30T17:54:59.008700+0000","last_undegraded":"2023-10-30T17:54:59.008700+0000","last_fullsized":"2023-10-30T17:54:59.008700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:21:36.073054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":855,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.609324+0000","last_change":"2023-10-30T17:33:40.644802+0000","last_active":"2023-10-30T17:54:54.609324+0000","last_peered":"2023-10-30T17:54:54.609324+0000","last_clean":"2023-10-30T17:54:54.609324+0000","last_became_active":"2023-10-30T17:33:40.584323+0000","last_became_peered":"2023-10-30T17:33:40.584323+0000","last_unstale":"2023-10-30T17:54:54.609324+0000","last_undegraded":"2023-10-30T17:54:54.609324+0000","last_fullsized":"2023-10-30T17:54:54.609324+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:41:03.792929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":860,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.906532+0000","last_change":"2023-10-30T17:32:34.658442+0000","last_active":"2023-10-30T17:54:55.906532+0000","last_peered":"2023-10-30T17:54:55.906532+0000","last_clean":"2023-10-30T17:54:55.906532+0000","last_became_active":"2023-10-30T17:32:34.631494+0000","last_became_peered":"2023-10-30T17:32:34.631494+0000","last_unstale":"2023-10-30T17:54:55.906532+0000","last_undegraded":"2023-10-30T17:54:55.906532+0000","last_fullsized":"2023-10-30T17:54:55.906532+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:36:36.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":858,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.956509+0000","last_change":"2023-10-30T17:33:46.593988+0000","last_active":"2023-10-30T17:54:56.956509+0000","last_peered":"2023-10-30T17:54:56.956509+0000","last_clean":"2023-10-30T17:54:56.956509+0000","last_became_active":"2023-10-30T17:33:46.588555+0000","last_became_peered":"2023-10-30T17:33:46.588555+0000","last_unstale":"2023-10-30T17:54:56.956509+0000","last_undegraded":"2023-10-30T17:54:56.956509+0000","last_fullsized":"2023-10-30T17:54:56.956509+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:57.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.998017+0000","last_change":"2023-10-30T17:32:35.388556+0000","last_active":"2023-10-30T17:54:58.998017+0000","last_peered":"2023-10-30T17:54:58.998017+0000","last_clean":"2023-10-30T17:54:58.998017+0000","last_became_active":"2023-10-30T17:32:35.383414+0000","last_became_peered":"2023-10-30T17:32:35.383414+0000","last_unstale":"2023-10-30T17:54:58.998017+0000","last_undegraded":"2023-10-30T17:54:58.998017+0000","last_fullsized":"2023-10-30T17:54:58.998017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:46.189212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":871,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.595206+0000","last_change":"2023-10-30T17:33:38.185755+0000","last_active":"2023-10-30T17:54:54.595206+0000","last_peered":"2023-10-30T17:54:54.595206+0000","last_clean":"2023-10-30T17:54:54.595206+0000","last_became_active":"2023-10-30T17:33:38.176989+0000","last_became_peered":"2023-10-30T17:33:38.176989+0000","last_unstale":"2023-10-30T17:54:54.595206+0000","last_undegraded":"2023-10-30T17:54:54.595206+0000","last_fullsized":"2023-10-30T17:54:54.595206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:25:37.689974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":878,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.580413+0000","last_change":"2023-10-30T17:32:34.318569+0000","last_active":"2023-10-30T17:54:54.580413+0000","last_peered":"2023-10-30T17:54:54.580413+0000","last_clean":"2023-10-30T17:54:54.580413+0000","last_became_active":"2023-10-30T17:32:34.287031+0000","last_became_peered":"2023-10-30T17:32:34.287031+0000","last_unstale":"2023-10-30T17:54:54.580413+0000","last_undegraded":"2023-10-30T17:54:54.580413+0000","last_fullsized":"2023-10-30T17:54:54.580413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:55:59.080654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":870,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.610146+0000","last_change":"2023-10-30T17:33:41.123654+0000","last_active":"2023-10-30T17:54:54.610146+0000","last_peered":"2023-10-30T17:54:54.610146+0000","last_clean":"2023-10-30T17:54:54.610146+0000","last_became_active":"2023-10-30T17:33:41.116885+0000","last_became_peered":"2023-10-30T17:33:41.116885+0000","last_unstale":"2023-10-30T17:54:54.610146+0000","last_undegraded":"2023-10-30T17:54:54.610146+0000","last_fullsized":"2023-10-30T17:54:54.610146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:51:37.700398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.996395+0000","last_change":"2023-10-30T17:32:35.307241+0000","last_active":"2023-10-30T17:54:58.996395+0000","last_peered":"2023-10-30T17:54:58.996395+0000","last_clean":"2023-10-30T17:54:58.996395+0000","last_became_active":"2023-10-30T17:32:35.277163+0000","last_became_peered":"2023-10-30T17:32:35.277163+0000","last_unstale":"2023-10-30T17:54:58.996395+0000","last_undegraded":"2023-10-30T17:54:58.996395+0000","last_fullsized":"2023-10-30T17:54:58.996395+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:02.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":848,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.025231+0000","last_change":"2023-10-30T17:33:43.745646+0000","last_active":"2023-10-30T17:54:59.025231+0000","last_peered":"2023-10-30T17:54:59.025231+0000","last_clean":"2023-10-30T17:54:59.025231+0000","last_became_active":"2023-10-30T17:33:43.736021+0000","last_became_peered":"2023-10-30T17:33:43.736021+0000","last_unstale":"2023-10-30T17:54:59.025231+0000","last_undegraded":"2023-10-30T17:54:59.025231+0000","last_fullsized":"2023-10-30T17:54:59.025231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:29.825853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.021371+0000","last_change":"2023-10-30T17:32:35.628758+0000","last_active":"2023-10-30T17:54:59.021371+0000","last_peered":"2023-10-30T17:54:59.021371+0000","last_clean":"2023-10-30T17:54:59.021371+0000","last_became_active":"2023-10-30T17:32:35.623395+0000","last_became_peered":"2023-10-30T17:32:35.623395+0000","last_unstale":"2023-10-30T17:54:59.021371+0000","last_undegraded":"2023-10-30T17:54:59.021371+0000","last_fullsized":"2023-10-30T17:54:59.021371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:47:08.984576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":966,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.254775+0000","last_change":"2023-10-30T17:32:28.719757+0000","last_active":"2023-10-30T17:54:59.254775+0000","last_peered":"2023-10-30T17:54:59.254775+0000","last_clean":"2023-10-30T17:54:59.254775+0000","last_became_active":"2023-10-30T17:32:28.686958+0000","last_became_peered":"2023-10-30T17:32:28.686958+0000","last_unstale":"2023-10-30T17:54:59.254775+0000","last_undegraded":"2023-10-30T17:54:59.254775+0000","last_fullsized":"2023-10-30T17:54:59.254775+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:25.778908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:25.778908+0000","last_clean_scrub_stamp":"2023-10-30T17:32:25.778908+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:41:21.329138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":855,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.630478+0000","last_change":"2023-10-30T17:33:40.638676+0000","last_active":"2023-10-30T17:54:54.630478+0000","last_peered":"2023-10-30T17:54:54.630478+0000","last_clean":"2023-10-30T17:54:54.630478+0000","last_became_active":"2023-10-30T17:33:40.602436+0000","last_became_peered":"2023-10-30T17:33:40.602436+0000","last_unstale":"2023-10-30T17:54:54.630478+0000","last_undegraded":"2023-10-30T17:54:54.630478+0000","last_fullsized":"2023-10-30T17:54:54.630478+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:41.628722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":849,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.013211+0000","last_change":"2023-10-30T17:33:38.623594+0000","last_active":"2023-10-30T17:54:59.013211+0000","last_peered":"2023-10-30T17:54:59.013211+0000","last_clean":"2023-10-30T17:54:59.013211+0000","last_became_active":"2023-10-30T17:33:38.616054+0000","last_became_peered":"2023-10-30T17:33:38.616054+0000","last_unstale":"2023-10-30T17:54:59.013211+0000","last_undegraded":"2023-10-30T17:54:59.013211+0000","last_fullsized":"2023-10-30T17:54:59.013211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:05:44.520215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":854,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.272308+0000","last_change":"2023-10-30T17:33:40.742636+0000","last_active":"2023-10-30T17:54:59.272308+0000","last_peered":"2023-10-30T17:54:59.272308+0000","last_clean":"2023-10-30T17:54:59.272308+0000","last_became_active":"2023-10-30T17:33:40.735511+0000","last_became_peered":"2023-10-30T17:33:40.735511+0000","last_unstale":"2023-10-30T17:54:59.272308+0000","last_undegraded":"2023-10-30T17:54:59.272308+0000","last_fullsized":"2023-10-30T17:54:59.272308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:30.315904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":854,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.905041+0000","last_change":"2023-10-30T17:33:39.276657+0000","last_active":"2023-10-30T17:54:55.905041+0000","last_peered":"2023-10-30T17:54:55.905041+0000","last_clean":"2023-10-30T17:54:55.905041+0000","last_became_active":"2023-10-30T17:33:39.228561+0000","last_became_peered":"2023-10-30T17:33:39.228561+0000","last_unstale":"2023-10-30T17:54:55.905041+0000","last_undegraded":"2023-10-30T17:54:55.905041+0000","last_fullsized":"2023-10-30T17:54:55.905041+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:01.276873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":870,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.597303+0000","last_change":"2023-10-30T17:33:41.611205+0000","last_active":"2023-10-30T17:54:54.597303+0000","last_peered":"2023-10-30T17:54:54.597303+0000","last_clean":"2023-10-30T17:54:54.597303+0000","last_became_active":"2023-10-30T17:33:41.605208+0000","last_became_peered":"2023-10-30T17:33:41.605208+0000","last_unstale":"2023-10-30T17:54:54.597303+0000","last_undegraded":"2023-10-30T17:54:54.597303+0000","last_fullsized":"2023-10-30T17:54:54.597303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:09.587584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.980989+0000","last_change":"2023-10-30T17:33:44.441890+0000","last_active":"2023-10-30T17:54:57.980989+0000","last_peered":"2023-10-30T17:54:57.980989+0000","last_clean":"2023-10-30T17:54:57.980989+0000","last_became_active":"2023-10-30T17:33:44.434104+0000","last_became_peered":"2023-10-30T17:33:44.434104+0000","last_unstale":"2023-10-30T17:54:57.980989+0000","last_undegraded":"2023-10-30T17:54:57.980989+0000","last_fullsized":"2023-10-30T17:54:57.980989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:11.365044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":855,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.899091+0000","last_change":"2023-10-30T17:33:43.330759+0000","last_active":"2023-10-30T17:54:55.899091+0000","last_peered":"2023-10-30T17:54:55.899091+0000","last_clean":"2023-10-30T17:54:55.899091+0000","last_became_active":"2023-10-30T17:33:43.322860+0000","last_became_peered":"2023-10-30T17:33:43.322860+0000","last_unstale":"2023-10-30T17:54:55.899091+0000","last_undegraded":"2023-10-30T17:54:55.899091+0000","last_fullsized":"2023-10-30T17:54:55.899091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:56:01.616064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.020934+0000","last_change":"2023-10-30T17:33:53.140054+0000","last_active":"2023-10-30T17:54:58.020934+0000","last_peered":"2023-10-30T17:54:58.020934+0000","last_clean":"2023-10-30T17:54:58.020934+0000","last_became_active":"2023-10-30T17:33:53.130615+0000","last_became_peered":"2023-10-30T17:33:53.130615+0000","last_unstale":"2023-10-30T17:54:58.020934+0000","last_undegraded":"2023-10-30T17:54:58.020934+0000","last_fullsized":"2023-10-30T17:54:58.020934+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:11.025225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":854,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.244856+0000","last_change":"2023-10-30T17:33:37.571189+0000","last_active":"2023-10-30T17:54:59.244856+0000","last_peered":"2023-10-30T17:54:59.244856+0000","last_clean":"2023-10-30T17:54:59.244856+0000","last_became_active":"2023-10-30T17:33:37.540749+0000","last_became_peered":"2023-10-30T17:33:37.540749+0000","last_unstale":"2023-10-30T17:54:59.244856+0000","last_undegraded":"2023-10-30T17:54:59.244856+0000","last_fullsized":"2023-10-30T17:54:59.244856+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:42.432458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":848,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.016385+0000","last_change":"2023-10-30T17:33:58.392321+0000","last_active":"2023-10-30T17:54:59.016385+0000","last_peered":"2023-10-30T17:54:59.016385+0000","last_clean":"2023-10-30T17:54:59.016385+0000","last_became_active":"2023-10-30T17:33:58.383320+0000","last_became_peered":"2023-10-30T17:33:58.383320+0000","last_unstale":"2023-10-30T17:54:59.016385+0000","last_undegraded":"2023-10-30T17:54:59.016385+0000","last_fullsized":"2023-10-30T17:54:59.016385+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:05.698087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":855,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.949358+0000","last_change":"2023-10-30T17:33:38.595237+0000","last_active":"2023-10-30T17:54:56.949358+0000","last_peered":"2023-10-30T17:54:56.949358+0000","last_clean":"2023-10-30T17:54:56.949358+0000","last_became_active":"2023-10-30T17:33:38.589479+0000","last_became_peered":"2023-10-30T17:33:38.589479+0000","last_unstale":"2023-10-30T17:54:56.949358+0000","last_undegraded":"2023-10-30T17:54:56.949358+0000","last_fullsized":"2023-10-30T17:54:56.949358+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:17:44.312113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":857,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.959591+0000","last_change":"2023-10-30T17:33:40.505225+0000","last_active":"2023-10-30T17:54:56.959591+0000","last_peered":"2023-10-30T17:54:56.959591+0000","last_clean":"2023-10-30T17:54:56.959591+0000","last_became_active":"2023-10-30T17:33:40.476315+0000","last_became_peered":"2023-10-30T17:33:40.476315+0000","last_unstale":"2023-10-30T17:54:56.959591+0000","last_undegraded":"2023-10-30T17:54:56.959591+0000","last_fullsized":"2023-10-30T17:54:56.959591+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:57:02.608925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":848,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.006691+0000","last_change":"2023-10-30T17:33:38.803571+0000","last_active":"2023-10-30T17:54:59.006691+0000","last_peered":"2023-10-30T17:54:59.006691+0000","last_clean":"2023-10-30T17:54:59.006691+0000","last_became_active":"2023-10-30T17:33:38.795801+0000","last_became_peered":"2023-10-30T17:33:38.795801+0000","last_unstale":"2023-10-30T17:54:59.006691+0000","last_undegraded":"2023-10-30T17:54:59.006691+0000","last_fullsized":"2023-10-30T17:54:59.006691+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:40:52.889892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":17,"seq":73014444302,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60904,"kb_used_data":7512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310936,"statfs":{"total":96636764160,"available":96574398464,"internally_reserved":0,"allocated":7692288,"data_stored":4265037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":7.8239999999999998,"5min":14.673999999999999,"15min":20.202999999999999},"min":{"1min":4.2460000000000004,"5min":4.2460000000000004,"15min":4.2460000000000004},"max":{"1min":15.343,"5min":217.97399999999999,"15min":296.49400000000003},"last":6.1900000000000004},{"interface":"front","average":{"1min":13.112,"5min":16.077000000000002,"15min":21.355},"min":{"1min":7.6120000000000001,"5min":5.6189999999999998,"15min":4.6299999999999999},"max":{"1min":31.678999999999998,"5min":134.833,"15min":276.49799999999999},"last":8.9879999999999995}]},{"osd":2,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":16.689,"5min":19.114999999999998,"15min":26.032},"min":{"1min":6.319,"5min":5.915,"15min":5.915},"max":{"1min":101.64100000000001,"5min":218.22399999999999,"15min":591.02499999999998},"last":11.734},{"interface":"front","average":{"1min":19.178999999999998,"5min":20.963000000000001,"15min":25.277000000000001},"min":{"1min":9.0850000000000009,"5min":7.6520000000000001,"15min":6.4930000000000003},"max":{"1min":103.506,"5min":228.36500000000001,"15min":259.52300000000002},"last":10.427}]},{"osd":3,"last update":"Mon Oct 30 17:53:58 2023","interfaces":[{"interface":"back","average":{"1min":23.952999999999999,"5min":78.492000000000004,"15min":43.039000000000001},"min":{"1min":6.9329999999999998,"5min":6.6879999999999997,"15min":5.8099999999999996},"max":{"1min":282.685,"5min":5036.1790000000001,"15min":5036.1790000000001},"last":9.7240000000000002},{"interface":"front","average":{"1min":24.942,"5min":81.421000000000006,"15min":47.087000000000003},"min":{"1min":8.4399999999999995,"5min":7.2770000000000001,"15min":6.4909999999999997},"max":{"1min":190.00899999999999,"5min":5057.1229999999996,"15min":5057.1229999999996},"last":13.465999999999999}]},{"osd":4,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":24.963000000000001,"5min":22.420999999999999,"15min":22.024000000000001},"min":{"1min":7.3259999999999996,"5min":7.3259999999999996,"15min":6.3689999999999998},"max":{"1min":159.94399999999999,"5min":204.91,"15min":281.16399999999999},"last":16.094000000000001},{"interface":"front","average":{"1min":29.393999999999998,"5min":26.954000000000001,"15min":27.579000000000001},"min":{"1min":9.0530000000000008,"5min":9.0530000000000008,"15min":5.8879999999999999},"max":{"1min":190.24199999999999,"5min":218.85300000000001,"15min":297.57299999999998},"last":14.125}]},{"osd":5,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":19.146999999999998,"5min":25.224,"15min":28.202999999999999},"min":{"1min":8.6660000000000004,"5min":8.4960000000000004,"15min":7.9370000000000003},"max":{"1min":44.509,"5min":260.81099999999998,"15min":260.81099999999998},"last":18.850000000000001},{"interface":"front","average":{"1min":15.294,"5min":22.741,"15min":25.870000000000001},"min":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.1189999999999998},"max":{"1min":32.457999999999998,"5min":236.83500000000001,"15min":236.83500000000001},"last":16.760999999999999}]},{"osd":6,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":19.766999999999999,"5min":22.66,"15min":27.329000000000001},"min":{"1min":10.051,"5min":9.968,"15min":8.4019999999999992},"max":{"1min":37.848999999999997,"5min":125.158,"15min":292.47500000000002},"last":17.888000000000002},{"interface":"front","average":{"1min":20.893000000000001,"5min":26.652000000000001,"15min":28.780000000000001},"min":{"1min":11.971,"5min":6.8449999999999998,"15min":6.8449999999999998},"max":{"1min":62.518000000000001,"5min":189.01599999999999,"15min":273.64999999999998},"last":15.422000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":22.042999999999999,"5min":25.934999999999999,"15min":29.561},"min":{"1min":12.084,"5min":10.622999999999999,"15min":9.157},"max":{"1min":60.759,"5min":204.12200000000001,"15min":262.74200000000002},"last":12.778},{"interface":"front","average":{"1min":26.411999999999999,"5min":31.727,"15min":33.009999999999998},"min":{"1min":13.321,"5min":12.188000000000001,"15min":10.122999999999999},"max":{"1min":100.953,"5min":219.833,"15min":264.81},"last":19.789000000000001}]}]},{"osd":3,"up_from":16,"seq":68719476998,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83092,"kb_used_data":7492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288748,"statfs":{"total":96636764160,"available":96551677952,"internally_reserved":0,"allocated":7671808,"data_stored":4216833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":9.1400000000000006,"5min":16.59,"15min":16.803999999999998},"min":{"1min":4.734,"5min":4.3869999999999996,"15min":4.3869999999999996},"max":{"1min":19.263000000000002,"5min":203.85499999999999,"15min":271.12799999999999},"last":9.2040000000000006},{"interface":"front","average":{"1min":12.346,"5min":17.565999999999999,"15min":18.774000000000001},"min":{"1min":5.5490000000000004,"5min":4.8449999999999998,"15min":4.367},"max":{"1min":36.253,"5min":176.83500000000001,"15min":364.05399999999997},"last":11.433999999999999}]},{"osd":1,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":19.100000000000001,"5min":22.856999999999999,"15min":24.716000000000001},"min":{"1min":7.3470000000000004,"5min":6.6589999999999998,"15min":5.5380000000000003},"max":{"1min":74.484999999999999,"5min":178.04300000000001,"15min":314.19099999999997},"last":19.881},{"interface":"front","average":{"1min":24.044,"5min":23.404,"15min":23.681000000000001},"min":{"1min":6.5609999999999999,"5min":5.9290000000000003,"15min":5.609},"max":{"1min":159.42099999999999,"5min":210.381,"15min":240.64500000000001},"last":13.476000000000001}]},{"osd":2,"last update":"Mon Oct 30 17:54:19 2023","interfaces":[{"interface":"back","average":{"1min":34.308999999999997,"5min":27.562000000000001,"15min":24.027999999999999},"min":{"1min":7.2969999999999997,"5min":7.2489999999999997,"15min":5.234},"max":{"1min":251.941,"5min":251.941,"15min":397.50099999999998},"last":17.768999999999998},{"interface":"front","average":{"1min":19.899000000000001,"5min":22.161999999999999,"15min":25.760999999999999},"min":{"1min":7.1539999999999999,"5min":6.54,"15min":6.54},"max":{"1min":85.495000000000005,"5min":147.50399999999999,"15min":219.65000000000001},"last":10.699}]},{"osd":4,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":22.57,"5min":33.298000000000002,"15min":27.831},"min":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":5.8140000000000001},"max":{"1min":164.642,"5min":318.92200000000003,"15min":400.13999999999999},"last":12.052},{"interface":"front","average":{"1min":17.919,"5min":25.715,"15min":22.006},"min":{"1min":8.3650000000000002,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":83.346999999999994,"5min":268.01600000000002,"15min":325.846},"last":18.446999999999999}]},{"osd":5,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":20.472000000000001,"5min":21.146999999999998,"15min":23.655000000000001},"min":{"1min":9.2029999999999994,"5min":7.7569999999999997,"15min":7.7569999999999997},"max":{"1min":84.305000000000007,"5min":149.61600000000001,"15min":221.821},"last":16.879000000000001},{"interface":"front","average":{"1min":30.245000000000001,"5min":27.282,"15min":28.794},"min":{"1min":9.75,"5min":7.2009999999999996,"15min":7.2009999999999996},"max":{"1min":225.15299999999999,"5min":225.15299999999999,"15min":372.13999999999999},"last":15.724}]},{"osd":6,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":32.417999999999999,"5min":30.308,"15min":28.475000000000001},"min":{"1min":10.678000000000001,"5min":7.9589999999999996,"15min":7.7439999999999998},"max":{"1min":272.536,"5min":272.536,"15min":360.26999999999998},"last":58.084000000000003},{"interface":"front","average":{"1min":23.282,"5min":23.863,"15min":25.344000000000001},"min":{"1min":11.023999999999999,"5min":9.3040000000000003,"15min":7.524},"max":{"1min":113.96599999999999,"5min":162.61799999999999,"15min":336.68700000000001},"last":97.793000000000006}]},{"osd":7,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":27.902999999999999,"5min":31.673999999999999,"15min":29.501000000000001},"min":{"1min":12.977,"5min":9.8399999999999999,"15min":9.7059999999999995},"max":{"1min":178.43100000000001,"5min":196.05699999999999,"15min":281.15699999999998},"last":55.932000000000002},{"interface":"front","average":{"1min":31.780999999999999,"5min":30.504999999999999,"15min":29.385999999999999},"min":{"1min":11.692,"5min":10.587999999999999,"15min":10.528},"max":{"1min":177.44900000000001,"5min":182.07499999999999,"15min":233.489},"last":31.356999999999999}]}]},{"osd":1,"up_from":16,"seq":68719477009,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61300,"kb_used_data":7904,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310540,"statfs":{"total":96636764160,"available":96573992960,"internally_reserved":0,"allocated":8093696,"data_stored":4628450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":10.568,"5min":13.151999999999999,"15min":17.620999999999999},"min":{"1min":5.1379999999999999,"5min":4.8099999999999996,"15min":3.8279999999999998},"max":{"1min":26.812000000000001,"5min":162.364,"15min":313.69400000000002},"last":11.885999999999999},{"interface":"front","average":{"1min":13.189,"5min":13.273999999999999,"15min":17.713000000000001},"min":{"1min":8.4429999999999996,"5min":5.4560000000000004,"15min":4.9089999999999998},"max":{"1min":21.195,"5min":59.277000000000001,"15min":209.62799999999999},"last":8.4979999999999993}]},{"osd":2,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":17.059999999999999,"5min":16.346,"15min":21.375},"min":{"1min":8.5259999999999998,"5min":6.2190000000000003,"15min":6.0279999999999996},"max":{"1min":84.930999999999997,"5min":95.995000000000005,"15min":267.83499999999998},"last":14.233000000000001},{"interface":"front","average":{"1min":19.794,"5min":18.084,"15min":20.032},"min":{"1min":11.779999999999999,"5min":7.0270000000000001,"15min":5.8550000000000004},"max":{"1min":89.875,"5min":89.875,"15min":178.44},"last":26.521000000000001}]},{"osd":3,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":10.504,"5min":246.75800000000001,"15min":96.697000000000003},"min":{"1min":6.1180000000000003,"5min":6.1050000000000004,"15min":5.2990000000000004},"max":{"1min":31.059000000000001,"5min":8788.2559999999994,"15min":8788.2559999999994},"last":8.9199999999999999},{"interface":"front","average":{"1min":17.736999999999998,"5min":247.22,"15min":98.808000000000007},"min":{"1min":11.208,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":29.475999999999999,"5min":8677.7369999999992,"15min":8677.7369999999992},"last":29.404}]},{"osd":4,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":15.772,"5min":17.181999999999999,"15min":23.558},"min":{"1min":7.7359999999999998,"5min":6.8220000000000001,"15min":6.8220000000000001},"max":{"1min":91.501999999999995,"5min":91.501999999999995,"15min":235.125},"last":9.5410000000000004},{"interface":"front","average":{"1min":20.797999999999998,"5min":21.347000000000001,"15min":23.934000000000001},"min":{"1min":10.278,"5min":7.673,"15min":7.1829999999999998},"max":{"1min":92.772000000000006,"5min":92.772000000000006,"15min":334.19900000000001},"last":13.321}]},{"osd":5,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":15.865,"5min":19.510000000000002,"15min":27.055},"min":{"1min":9.0389999999999997,"5min":9.0389999999999997,"15min":7.6360000000000001},"max":{"1min":43.468000000000004,"5min":89.837000000000003,"15min":220.65799999999999},"last":28.439},{"interface":"front","average":{"1min":21.504999999999999,"5min":21.619,"15min":31.059999999999999},"min":{"1min":14.56,"5min":8.0869999999999997,"15min":8.0869999999999997},"max":{"1min":50.332000000000001,"5min":134.42599999999999,"15min":262.61799999999999},"last":32.908000000000001}]},{"osd":6,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":10.253,"5min":15.454000000000001,"15min":24.402000000000001},"min":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":6.8769999999999998},"max":{"1min":20.927,"5min":200.369,"15min":249.25299999999999},"last":16.367000000000001},{"interface":"front","average":{"1min":17.808,"5min":20.117999999999999,"15min":21.41},"min":{"1min":10.965,"5min":8.8610000000000007,"15min":6.2880000000000003},"max":{"1min":28.323,"5min":201.756,"15min":259.03899999999999},"last":31.189}]},{"osd":7,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":17.879000000000001,"5min":21.974,"15min":27.113},"min":{"1min":11.935,"5min":9.8200000000000003,"15min":9.4269999999999996},"max":{"1min":29.427,"5min":100.73999999999999,"15min":200.62100000000001},"last":25.783000000000001},{"interface":"front","average":{"1min":21.350999999999999,"5min":26.129999999999999,"15min":30.661000000000001},"min":{"1min":16.481000000000002,"5min":10.682,"15min":10.115},"max":{"1min":36.606999999999999,"5min":203.929,"15min":205.61799999999999},"last":32.268000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509712,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60844,"kb_used_data":7460,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310996,"statfs":{"total":96636764160,"available":96574459904,"internally_reserved":0,"allocated":7639040,"data_stored":4202947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":14.316000000000001,"5min":13.782,"15min":18.59},"min":{"1min":5.3460000000000001,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":106.839,"5min":106.839,"15min":334.36900000000003},"last":6.2960000000000003},{"interface":"front","average":{"1min":18.443000000000001,"5min":17.509,"15min":21.878},"min":{"1min":7.7990000000000004,"5min":7.1319999999999997,"15min":6.7489999999999997},"max":{"1min":111.07599999999999,"5min":111.07599999999999,"15min":283.46899999999999},"last":11.449}]},{"osd":1,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":25.234000000000002,"5min":23.196999999999999,"15min":24.32},"min":{"1min":9.1259999999999994,"5min":6.1539999999999999,"15min":4.7830000000000004},"max":{"1min":122.607,"5min":194.499,"15min":273.93599999999998},"last":12.382999999999999},{"interface":"front","average":{"1min":18.93,"5min":14.327999999999999,"15min":15.942},"min":{"1min":5.0069999999999997,"5min":4.9020000000000001,"15min":4.1890000000000001},"max":{"1min":109.767,"5min":109.767,"15min":152.87299999999999},"last":7.9729999999999999}]},{"osd":3,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":12.736000000000001,"5min":219.62799999999999,"15min":85.793999999999997},"min":{"1min":5.6429999999999998,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":62.838000000000001,"5min":8749.8639999999996,"15min":8749.8639999999996},"last":108.666},{"interface":"front","average":{"1min":19.050999999999998,"5min":225.965,"15min":91.936000000000007},"min":{"1min":9.2469999999999999,"5min":7.7859999999999996,"15min":6.6799999999999997},"max":{"1min":75.923000000000002,"5min":8752.7450000000008,"15min":8752.7450000000008},"last":109.779}]},{"osd":4,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":24.823,"5min":22.719000000000001,"15min":28.533999999999999},"min":{"1min":8.8420000000000005,"5min":8.5990000000000002,"15min":8.5990000000000002},"max":{"1min":115.70999999999999,"5min":124.502,"15min":301.34399999999999},"last":14.112},{"interface":"front","average":{"1min":27.699999999999999,"5min":27.585000000000001,"15min":32.787999999999997},"min":{"1min":11.974,"5min":11.974,"15min":11.824},"max":{"1min":117.285,"5min":134.44800000000001,"15min":300.68900000000002},"last":15.726000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":21.994,"5min":19.698,"15min":29.402999999999999},"min":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":124.914,"5min":198.46700000000001,"15min":318.01900000000001},"last":25.18},{"interface":"front","average":{"1min":21.603999999999999,"5min":18.942,"15min":24.146000000000001},"min":{"1min":10.16,"5min":7.5960000000000001,"15min":5.8920000000000003},"max":{"1min":122.861,"5min":122.861,"15min":325.87099999999998},"last":107.361}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":19.571000000000002,"5min":18.742000000000001,"15min":20.893000000000001},"min":{"1min":8.9290000000000003,"5min":8.7420000000000009,"15min":6.3979999999999997},"max":{"1min":70.927999999999997,"5min":70.927999999999997,"15min":328.983},"last":8.9909999999999997},{"interface":"front","average":{"1min":25.178000000000001,"5min":26.393999999999998,"15min":30.449000000000002},"min":{"1min":11.391,"5min":11.391,"15min":10.986000000000001},"max":{"1min":124.36799999999999,"5min":133.02199999999999,"15min":306.81799999999998},"last":17.527000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":17.791,"5min":16.997,"15min":22.556000000000001},"min":{"1min":7.8630000000000004,"5min":7.2549999999999999,"15min":6.9669999999999996},"max":{"1min":127.804,"5min":127.804,"15min":307.69499999999999},"last":9.9990000000000006},{"interface":"front","average":{"1min":21.841999999999999,"5min":22.962,"15min":28.602},"min":{"1min":10.927,"5min":8.9550000000000001,"15min":8.5700000000000003},"max":{"1min":129.26599999999999,"5min":129.26599999999999,"15min":317.13400000000001},"last":10.691000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83092,"kb_used_data":7492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288748,"statfs":{"total":96636764160,"available":96551677952,"internally_reserved":0,"allocated":7671808,"data_stored":4240466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77396427},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":18.553000000000001,"5min":17.256,"15min":18.238},"min":{"1min":4.6230000000000002,"5min":4.6230000000000002,"15min":4.1580000000000004},"max":{"1min":172.93199999999999,"5min":179.608,"15min":259.12200000000001},"last":5.548},{"interface":"front","average":{"1min":19.984999999999999,"5min":18.443999999999999,"15min":18.128},"min":{"1min":4.5999999999999996,"5min":4.5999999999999996,"15min":4.3959999999999999},"max":{"1min":176.59399999999999,"5min":176.59399999999999,"15min":258.25799999999998},"last":7.1600000000000001}]},{"osd":1,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":13.958,"5min":15.481,"15min":18.420999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.2939999999999996},"max":{"1min":50.183,"5min":83.355000000000004,"15min":249.69900000000001},"last":16.523},{"interface":"front","average":{"1min":16.277000000000001,"5min":19.504000000000001,"15min":24.225999999999999},"min":{"1min":8.0890000000000004,"5min":8.0640000000000001,"15min":5.0439999999999996},"max":{"1min":67.171000000000006,"5min":199.15299999999999,"15min":334.67599999999999},"last":15.115}]},{"osd":2,"last update":"Mon Oct 30 17:54:03 2023","interfaces":[{"interface":"back","average":{"1min":18.440999999999999,"5min":18.277999999999999,"15min":22.067},"min":{"1min":7.282,"5min":6.3410000000000002,"15min":6.3140000000000001},"max":{"1min":107.964,"5min":128.202,"15min":403.90899999999999},"last":17.457000000000001},{"interface":"front","average":{"1min":33.597000000000001,"5min":25.751999999999999,"15min":27.367999999999999},"min":{"1min":9.8949999999999996,"5min":8.4559999999999995,"15min":7.6829999999999998},"max":{"1min":276.67700000000002,"5min":276.67700000000002,"15min":354.34800000000001},"last":25.731000000000002}]},{"osd":3,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":19.672000000000001,"5min":165.75800000000001,"15min":72.480000000000004},"min":{"1min":8.8859999999999992,"5min":8.8859999999999992,"15min":6.226},"max":{"1min":71.772999999999996,"5min":8740.9120000000003,"15min":8740.9120000000003},"last":23.808},{"interface":"front","average":{"1min":16.125,"5min":162.107,"15min":70.201999999999998},"min":{"1min":7.444,"5min":7.3840000000000003,"15min":6.1120000000000001},"max":{"1min":50.947000000000003,"5min":8827.2800000000007,"15min":8827.2800000000007},"last":21.263000000000002}]},{"osd":4,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":17.946999999999999,"5min":23.007999999999999,"15min":24.891999999999999},"min":{"1min":6.758,"5min":6.758,"15min":6.758},"max":{"1min":73.082999999999998,"5min":193.517,"15min":265.78300000000002},"last":20.126999999999999},{"interface":"front","average":{"1min":25.338000000000001,"5min":28.114999999999998,"15min":26.849},"min":{"1min":11.759,"5min":10.228999999999999,"15min":8.5839999999999996},"max":{"1min":121.331,"5min":204.27799999999999,"15min":297.99900000000002},"last":26.582000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:54:03 2023","interfaces":[{"interface":"back","average":{"1min":21.972000000000001,"5min":25.536000000000001,"15min":30.370000000000001},"min":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":7.7729999999999997},"max":{"1min":96.088999999999999,"5min":235.08500000000001,"15min":396.77300000000002},"last":15.843999999999999},{"interface":"front","average":{"1min":20.291,"5min":21.542000000000002,"15min":25.526},"min":{"1min":8.0500000000000007,"5min":6.5010000000000003,"15min":5.5010000000000003},"max":{"1min":103.301,"5min":266.30200000000002,"15min":333.88200000000001},"last":21.800999999999998}]},{"osd":6,"last update":"Mon Oct 30 17:54:03 2023","interfaces":[{"interface":"back","average":{"1min":22.289000000000001,"5min":23.263000000000002,"15min":25.934999999999999},"min":{"1min":9.2609999999999992,"5min":9.1809999999999992,"15min":6.9400000000000004},"max":{"1min":68.593999999999994,"5min":174.517,"15min":445.44799999999998},"last":23.141999999999999},{"interface":"front","average":{"1min":31.324999999999999,"5min":28.215,"15min":29.988},"min":{"1min":13.523,"5min":10.054,"15min":8.0060000000000002},"max":{"1min":100.619,"5min":205.42500000000001,"15min":400.87599999999998},"last":22.446999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542418,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78916,"kb_used_data":7476,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292924,"statfs":{"total":96636764160,"available":96555954176,"internally_reserved":0,"allocated":7655424,"data_stored":4210307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":9.3670000000000009,"5min":16.529,"15min":18.324000000000002},"min":{"1min":4.7869999999999999,"5min":4.3339999999999996,"15min":4.1840000000000002},"max":{"1min":26.079000000000001,"5min":135.98099999999999,"15min":254.899},"last":12.679},{"interface":"front","average":{"1min":11.316000000000001,"5min":17.039999999999999,"15min":19.542000000000002},"min":{"1min":6.4569999999999999,"5min":5.4219999999999997,"15min":5.4219999999999997},"max":{"1min":28.175999999999998,"5min":97.100999999999999,"15min":270.51600000000002},"last":16.597000000000001}]},{"osd":1,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":19.887,"5min":20.942,"15min":20.899999999999999},"min":{"1min":7.5099999999999998,"5min":7.2350000000000003,"15min":5.1120000000000001},"max":{"1min":138.97300000000001,"5min":149.53899999999999,"15min":177.15799999999999},"last":7.3620000000000001},{"interface":"front","average":{"1min":12.612,"5min":20.626000000000001,"15min":23.800999999999998},"min":{"1min":7.5529999999999999,"5min":6.2199999999999998,"15min":5.883},"max":{"1min":28.498000000000001,"5min":221.483,"15min":309.04199999999997},"last":13.871}]},{"osd":2,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":14.561999999999999,"5min":19.271000000000001,"15min":20.66},"min":{"1min":7.1379999999999999,"5min":6.8929999999999998,"15min":5.859},"max":{"1min":31.204000000000001,"5min":122.018,"15min":231.15600000000001},"last":14.536},{"interface":"front","average":{"1min":16.201000000000001,"5min":24.071999999999999,"15min":23.096},"min":{"1min":11.792,"5min":11.173999999999999,"15min":6.9889999999999999},"max":{"1min":26.43,"5min":149.17500000000001,"15min":197.58699999999999},"last":15.353}]},{"osd":3,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":9.8900000000000006,"5min":253.59399999999999,"15min":99.061000000000007},"min":{"1min":6.6689999999999996,"5min":5.5039999999999996,"15min":5.3849999999999998},"max":{"1min":14.733000000000001,"5min":7958.2420000000002,"15min":7958.2420000000002},"last":5.9400000000000004},{"interface":"front","average":{"1min":13.827,"5min":254.506,"15min":99.292000000000002},"min":{"1min":8.7859999999999996,"5min":6.7009999999999996,"15min":5.9859999999999998},"max":{"1min":23.07,"5min":7971.3900000000003,"15min":7971.3900000000003},"last":8.3620000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":15.43,"5min":25.451000000000001,"15min":25.949000000000002},"min":{"1min":10.375999999999999,"5min":7.5300000000000002,"15min":7.5300000000000002},"max":{"1min":22.405000000000001,"5min":236.994,"15min":262.15699999999998},"last":17.315999999999999},{"interface":"front","average":{"1min":16.157,"5min":23.646999999999998,"15min":23.992000000000001},"min":{"1min":9.6839999999999993,"5min":8.6479999999999997,"15min":6.7859999999999996},"max":{"1min":23.186,"5min":228.88200000000001,"15min":228.88200000000001},"last":10.827999999999999}]},{"osd":5,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":11.866,"5min":20.151,"15min":24.649999999999999},"min":{"1min":6.5540000000000003,"5min":6.2359999999999998,"15min":5.7809999999999997},"max":{"1min":18.652000000000001,"5min":191.154,"15min":289.22699999999998},"last":8.9949999999999992},{"interface":"front","average":{"1min":17.635000000000002,"5min":25.885999999999999,"15min":27.363},"min":{"1min":14.632999999999999,"5min":10.679,"15min":7.835},"max":{"1min":25.047000000000001,"5min":230.613,"15min":230.613},"last":17.914999999999999}]},{"osd":7,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":27.414000000000001,"5min":24.706,"15min":26.167999999999999},"min":{"1min":9.2230000000000008,"5min":8.0370000000000008,"15min":7.2729999999999997},"max":{"1min":164.86600000000001,"5min":192.72399999999999,"15min":290.22300000000001},"last":10.035},{"interface":"front","average":{"1min":27.791,"5min":26.841000000000001,"15min":28.561},"min":{"1min":12.119999999999999,"5min":9.5440000000000005,"15min":7.4489999999999998},"max":{"1min":130.845,"5min":270.01900000000001,"15min":270.01900000000001},"last":15.973000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542418,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60948,"kb_used_data":7552,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310892,"statfs":{"total":96636764160,"available":96574353408,"internally_reserved":0,"allocated":7733248,"data_stored":4297155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":8.5139999999999993,"5min":21.638000000000002,"15min":19.297000000000001},"min":{"1min":4.5659999999999998,"5min":3.7290000000000001,"15min":3.7290000000000001},"max":{"1min":31.369,"5min":278.70299999999997,"15min":317.529},"last":8.1470000000000002},{"interface":"front","average":{"1min":8.5500000000000007,"5min":19.698,"15min":19.140999999999998},"min":{"1min":4.9850000000000003,"5min":4.9850000000000003,"15min":4.8040000000000003},"max":{"1min":32.368000000000002,"5min":174.08099999999999,"15min":200.29300000000001},"last":9.1690000000000005}]},{"osd":1,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":14.672000000000001,"5min":18,"15min":23.431999999999999},"min":{"1min":5.6029999999999998,"5min":5.6029999999999998,"15min":5.6029999999999998},"max":{"1min":25.289999999999999,"5min":136.81200000000001,"15min":307.86799999999999},"last":20.815999999999999},{"interface":"front","average":{"1min":12.586,"5min":18.454999999999998,"15min":23.405000000000001},"min":{"1min":7.9080000000000004,"5min":6.1669999999999998,"15min":6.0819999999999999},"max":{"1min":30.154,"5min":139.191,"15min":307.05500000000001},"last":6.8570000000000002}]},{"osd":2,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":14.393000000000001,"5min":19.882999999999999,"15min":24.52},"min":{"1min":7.2400000000000002,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":25.452999999999999,"5min":279.37099999999998,"15min":283.87599999999998},"last":112.958},{"interface":"front","average":{"1min":24.056000000000001,"5min":21.108000000000001,"15min":22.945},"min":{"1min":9.3710000000000004,"5min":6.9390000000000001,"15min":5.8259999999999996},"max":{"1min":198.02500000000001,"5min":198.02500000000001,"15min":321.54000000000002},"last":92.144999999999996}]},{"osd":3,"last update":"Mon Oct 30 17:54:12 2023","interfaces":[{"interface":"back","average":{"1min":27.785,"5min":165.78,"15min":72.799000000000007},"min":{"1min":9.952,"5min":5.9219999999999997,"15min":5.9219999999999997},"max":{"1min":247.03399999999999,"5min":8219.1820000000007,"15min":8219.1820000000007},"last":22.355},{"interface":"front","average":{"1min":14.07,"5min":161.76900000000001,"15min":71.828000000000003},"min":{"1min":8.7919999999999998,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":34.832999999999998,"5min":8140.4719999999998,"15min":8140.4719999999998},"last":25.198}]},{"osd":5,"last update":"Mon Oct 30 17:54:53 2023","interfaces":[{"interface":"back","average":{"1min":18.893000000000001,"5min":22.899000000000001,"15min":23.814},"min":{"1min":9.1790000000000003,"5min":6.4169999999999998,"15min":6.4169999999999998},"max":{"1min":65.632000000000005,"5min":292.22800000000001,"15min":292.22800000000001},"last":11.455},{"interface":"front","average":{"1min":18.199999999999999,"5min":22.989999999999998,"15min":22.789999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":59.558,"5min":179.67500000000001,"15min":180.28800000000001},"last":12.792999999999999}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":28.52,"5min":26.338999999999999,"15min":25.452000000000002},"min":{"1min":12.355,"5min":9.7889999999999997,"15min":6.5810000000000004},"max":{"1min":191.995,"5min":191.995,"15min":293.16300000000001},"last":12.082000000000001},{"interface":"front","average":{"1min":17.751999999999999,"5min":26.558,"15min":27.359999999999999},"min":{"1min":8.516,"5min":8.2420000000000009,"15min":7.6779999999999999},"max":{"1min":43.220999999999997,"5min":207.30099999999999,"15min":281.62799999999999},"last":23.100000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":18.670000000000002,"5min":26.806999999999999,"15min":28.308},"min":{"1min":12.002000000000001,"5min":8.0289999999999999,"15min":7.4969999999999999},"max":{"1min":43.93,"5min":396.928,"15min":396.928},"last":9.9209999999999994},{"interface":"front","average":{"1min":18.928000000000001,"5min":32.037999999999997,"15min":29.954999999999998},"min":{"1min":10.805999999999999,"5min":8.9809999999999999,"15min":7.0860000000000003},"max":{"1min":46.781999999999996,"5min":366.32499999999999,"15min":366.32499999999999},"last":21.715}]}]},{"osd":5,"up_from":13,"seq":55834575122,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79328,"kb_used_data":7888,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292512,"statfs":{"total":96636764160,"available":96555532288,"internally_reserved":0,"allocated":8077312,"data_stored":4637651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.609999999999999,"5min":14.541,"15min":16.334},"min":{"1min":4.7089999999999996,"5min":3.9910000000000001,"15min":3.9569999999999999},"max":{"1min":83.933999999999997,"5min":109.919,"15min":155.58799999999999},"last":6.1840000000000002},{"interface":"front","average":{"1min":17.244,"5min":16.513999999999999,"15min":18.204999999999998},"min":{"1min":7.4370000000000003,"5min":4.8789999999999996,"15min":4.6900000000000004},"max":{"1min":82.497,"5min":159.01900000000001,"15min":180.94499999999999},"last":15.596}]},{"osd":1,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":10.683999999999999,"5min":18.123999999999999,"15min":20.596},"min":{"1min":5.0970000000000004,"5min":5.0970000000000004,"15min":4.8810000000000002},"max":{"1min":30.02,"5min":228.15199999999999,"15min":233.22800000000001},"last":16.308},{"interface":"front","average":{"1min":14.571,"5min":15.516999999999999,"15min":18.213000000000001},"min":{"1min":8.5969999999999995,"5min":5.8259999999999996,"15min":5.1120000000000001},"max":{"1min":28.818999999999999,"5min":128.197,"15min":278.41300000000001},"last":18.452000000000002}]},{"osd":2,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":16.053999999999998,"5min":16.785,"15min":19.841000000000001},"min":{"1min":7.101,"5min":7.101,"15min":5.8949999999999996},"max":{"1min":30.600999999999999,"5min":135.75999999999999,"15min":234.09299999999999},"last":17.797000000000001},{"interface":"front","average":{"1min":13.555999999999999,"5min":15.548,"15min":20.547000000000001},"min":{"1min":6.657,"5min":6.5510000000000002,"15min":6.5060000000000002},"max":{"1min":27.719999999999999,"5min":87.316999999999993,"15min":277.52300000000002},"last":19.933}]},{"osd":3,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":13.231999999999999,"5min":151.136,"15min":65.766999999999996},"min":{"1min":6.2080000000000002,"5min":6.2080000000000002,"15min":6.2080000000000002},"max":{"1min":48.613999999999997,"5min":6785.9989999999998,"15min":6785.9989999999998},"last":16.978000000000002},{"interface":"front","average":{"1min":16.672999999999998,"5min":151.255,"15min":65.844999999999999},"min":{"1min":6.6870000000000003,"5min":6.6870000000000003,"15min":6.2839999999999998},"max":{"1min":54.656999999999996,"5min":6786.7420000000002,"15min":6786.7420000000002},"last":31.672000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.928999999999998,"15min":22.541},"min":{"1min":7.2590000000000003,"5min":6.9790000000000001,"15min":6.6609999999999996},"max":{"1min":24.433,"5min":143.75800000000001,"15min":286.96699999999998},"last":19.233000000000001},{"interface":"front","average":{"1min":18.093,"5min":23.826000000000001,"15min":26.565000000000001},"min":{"1min":12.166,"5min":12.166,"15min":8.609},"max":{"1min":26.777000000000001,"5min":144.934,"15min":276.17200000000003},"last":21.702000000000002}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.897,"5min":17.413,"15min":21.896000000000001},"min":{"1min":7.2969999999999997,"5min":6.5190000000000001,"15min":6.5190000000000001},"max":{"1min":77.432000000000002,"5min":77.432000000000002,"15min":280.36000000000001},"last":30.943999999999999},{"interface":"front","average":{"1min":16.088000000000001,"5min":21.170999999999999,"15min":26.117000000000001},"min":{"1min":9.4410000000000007,"5min":7.4020000000000001,"15min":7.4020000000000001},"max":{"1min":25.859999999999999,"5min":109.157,"15min":346.47000000000003},"last":32.313000000000002}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":17.742999999999999,"5min":21.178000000000001,"15min":28.771999999999998},"min":{"1min":8.8689999999999998,"5min":8.8689999999999998,"15min":7.6520000000000001},"max":{"1min":54.5,"5min":110.94799999999999,"15min":332.99099999999999},"last":170.19399999999999},{"interface":"front","average":{"1min":20.643000000000001,"5min":22.628,"15min":29.914999999999999},"min":{"1min":11.954000000000001,"5min":10.728,"15min":9.6750000000000007},"max":{"1min":72.408000000000001,"5min":145.00899999999999,"15min":332.64800000000002},"last":38.954000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T17:55:02.015 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T17:55:02.016 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T17:55:02.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.114+0000 7f2bc2c34640 10 monclient: get_monmap_and_config 2023-10-30T17:55:02.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.114+0000 7f2bc2c34640 10 monclient: build_initial_monmap 2023-10-30T17:55:02.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.114+0000 7f2bc2c34640 10 monclient: monmap: 2023-10-30T17:55:02.116 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:02.116 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:02.116 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:02.115328+0000 2023-10-30T17:55:02.117 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:02.115328+0000 2023-10-30T17:55:02.117 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:02.117 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.117 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:02.117 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:02.118 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:02.118 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:02.118 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:02.118 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 1 Processor -- start 2023-10-30T17:55:02.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 1 -- start start 2023-10-30T17:55:02.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 10 monclient: init 2023-10-30T17:55:02.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:02.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-10-30T17:55:02.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2bbc0b0ec0 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:02.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13f4f0 0x7f2bbc141c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 10 monclient(hunting): picked mon.noname-b con 0x7f2bbc13f4f0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:02.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2bbc06ceb0 0x7f2bbc1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 10 monclient(hunting): picked mon.noname-a con 0x7f2bbc06ceb0 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:02.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc06c370 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:02.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc0b12a0 con 0x7f2bbc06ceb0 2023-10-30T17:55:02.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:02.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc06c650 con 0x7f2bbc13f4f0 2023-10-30T17:55:02.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:02.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.115+0000 7f2bc2c34640 10 monclient(hunting): authenticate will time out at 2684.438797s 2023-10-30T17:55:02.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.116+0000 7f2bbbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13f4f0 0x7f2bbc141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.116+0000 7f2bc11d1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2bbc06ceb0 0x7f2bbc1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.116+0000 7f2bc09d0640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f2bbc0b0ec0 0x7f2bbc13ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59418/0 (socket says 172.21.15.170:59418) 2023-10-30T17:55:02.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.116+0000 7f2bc09d0640 1 -- 172.21.15.170:0/742378956 learned_addr learned my addr 172.21.15.170:0/742378956 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:02.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.116+0000 7f2bbbfff640 10 monclient(hunting): get_auth_request con 0x7f2bbc13f4f0 auth_method 0 2023-10-30T17:55:02.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.116+0000 7f2bbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:02.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.116+0000 7f2bbbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.116+0000 7f2bbbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.117+0000 7f2bc11d1640 10 monclient(hunting): get_auth_request con 0x7f2bbc06ceb0 auth_method 0 2023-10-30T17:55:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.117+0000 7f2bc11d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.117+0000 7f2bc11d1640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.117+0000 7f2bc11d1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.117+0000 7f2bbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:02.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.117+0000 7f2bbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:02.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.117+0000 7f2bbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:02.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.117+0000 7f2bc11d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:02.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.117+0000 7f2bc11d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:02.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.117+0000 7f2bc11d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:02.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.119+0000 7f2bbbfff640 10 monclient(hunting): handle_auth_done global_id 16301 payload 274 2023-10-30T17:55:02.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.119+0000 7f2bbbfff640 1 -- 172.21.15.170:0/742378956 >> v2:172.21.15.170:3300/0 conn(0x7f2bbc06ceb0 msgr2=0x7f2bbc1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:02.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.119+0000 7f2bbbfff640 1 --2- 172.21.15.170:0/742378956 >> v2:172.21.15.170:3300/0 conn(0x7f2bbc06ceb0 0x7f2bbc1421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.119+0000 7f2bbbfff640 1 -- 172.21.15.170:0/742378956 >> v1:172.21.15.170:6789/0 conn(0x7f2bbc0b0ec0 legacy=0x7f2bbc13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.119+0000 7f2bbbfff640 10 monclient: _finish_hunting 0 2023-10-30T17:55:02.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.119+0000 7f2bbbfff640 1 monclient: found mon.noname-b 2023-10-30T17:55:02.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.119+0000 7f2bbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:02.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.119+0000 7f2bbbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:55:02.130 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.119+0000 7f2bbbfff640 1 -- 172.21.15.170:0/742378956 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc0b1a70 con 0x7f2bbc13f4f0 2023-10-30T17:55:02.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.121+0000 7f2bbbfff640 1 --2- 172.21.15.170:0/742378956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13f4f0 0x7f2bbc141c90 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80097a0 tx=0x7f2ba802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6916227db40de4d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:02.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 1 -- 172.21.15.170:0/742378956 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba802e070 con 0x7f2bbc13f4f0 2023-10-30T17:55:02.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:02.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:02.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 10 monclient: dump: 2023-10-30T17:55:02.136 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:02.137 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:02.137 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.137 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.137 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:02.137 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.138 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:02.138 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:02.138 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:02.138 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 10 monclient: _finish_auth 0 2023-10-30T17:55:02.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:55:02.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:02.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 1 -- 172.21.15.170:0/742378956 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba80315a0 con 0x7f2bbc13f4f0 2023-10-30T17:55:02.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 5 monclient: authenticate success, global_id 16301 2023-10-30T17:55:02.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:02.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:02.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:02.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 1 -- 172.21.15.170:0/742378956 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8031930 con 0x7f2bbc13f4f0 2023-10-30T17:55:02.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:02.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 4 set_mon_vals no callback set 2023-10-30T17:55:02.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:55:02.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bbb7fe640 10 monclient: dump: 2023-10-30T17:55:02.141 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:02.142 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:02.142 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.142 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.143 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:02.143 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.143 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:02.143 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:02.144 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:02.144 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 10 monclient: shutdown 2023-10-30T17:55:02.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:02.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 1 -- 172.21.15.170:0/742378956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13f4f0 msgr2=0x7f2bbc141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 1 --2- 172.21.15.170:0/742378956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13f4f0 0x7f2bbc141c90 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80097a0 tx=0x7f2ba802b840 comp rx=0 tx=0).stop 2023-10-30T17:55:02.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 1 -- 172.21.15.170:0/742378956 shutdown_connections 2023-10-30T17:55:02.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 1 --2- 172.21.15.170:0/742378956 >> v2:172.21.15.170:3300/0 conn(0x7f2bbc06ceb0 0x7f2bbc1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 1 --2- 172.21.15.170:0/742378956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13f4f0 0x7f2bbc141c90 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 1 -- 172.21.15.170:0/742378956 >> 172.21.15.170:0/742378956 conn(0x7f2bbc0b1d40 msgr2=0x7f2bbc0b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:02.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 1 -- 172.21.15.170:0/742378956 shutdown_connections 2023-10-30T17:55:02.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.125+0000 7f2bc2c34640 1 -- 172.21.15.170:0/742378956 wait complete. 2023-10-30T17:55:02.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.126+0000 7f2bc2c34640 10 monclient: build_initial_monmap 2023-10-30T17:55:02.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.126+0000 7f2bc2c34640 10 monclient: monmap: 2023-10-30T17:55:02.147 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:02.147 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:02.147 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:02.147 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:02.148 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:02.148 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.148 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:02.148 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:02.148 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:02.149 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.126+0000 7f2bc2c34640 1 librados: starting msgr at 2023-10-30T17:55:02.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.126+0000 7f2bc2c34640 1 librados: starting objecter 2023-10-30T17:55:02.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.126+0000 7f2bc2c34640 1 Processor -- start 2023-10-30T17:55:02.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.126+0000 7f2bc2c34640 1 -- start start 2023-10-30T17:55:02.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.126+0000 7f2bc2c34640 1 librados: setting wanted keys 2023-10-30T17:55:02.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.126+0000 7f2bc2c34640 1 librados: calling monclient init 2023-10-30T17:55:02.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.126+0000 7f2bc2c34640 10 monclient: init 2023-10-30T17:55:02.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:02.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:55:02.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bbc06ceb0 0x7f2bbc13a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 10 monclient(hunting): picked mon.noname-b con 0x7f2bbc06ceb0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:02.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2bbc0b0ec0 0x7f2bbc1d6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 10 monclient(hunting): picked mon.noname-a con 0x7f2bbc0b0ec0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:02.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13a970 0x7f2bbc1d7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 10 monclient(hunting): picked mon.noname-c con 0x7f2bbc13a970 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:02.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:02.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc1d67e0 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:02.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc0af1c0 con 0x7f2bbc06ceb0 2023-10-30T17:55:02.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:02.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc0af380 con 0x7f2bbc13a970 2023-10-30T17:55:02.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:02.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.127+0000 7f2bc2c34640 10 monclient(hunting): authenticate will time out at 2684.450728s 2023-10-30T17:55:02.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc11d1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13a970 0x7f2bbc1d7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bbbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2bbc0b0ec0 0x7f2bbc1d6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bbbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2bbc0b0ec0 0x7f2bbc1d6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40388/0 (socket says 172.21.15.170:40388) 2023-10-30T17:55:02.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bbbfff640 1 -- 172.21.15.170:0/1563328916 learned_addr learned my addr 172.21.15.170:0/1563328916 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:02.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bbbfff640 10 monclient(hunting): get_auth_request con 0x7f2bbc0b0ec0 auth_method 0 2023-10-30T17:55:02.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:02.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bbbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bbbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc11d1640 10 monclient(hunting): get_auth_request con 0x7f2bbc13a970 auth_method 0 2023-10-30T17:55:02.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc11d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:02.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc11d1640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc11d1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc09d0640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bbc06ceb0 0x7f2bbc13a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc09d0640 10 monclient(hunting): get_auth_request con 0x7f2bbc06ceb0 auth_method 0 2023-10-30T17:55:02.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc09d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:02.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc09d0640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc09d0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc11d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:02.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc11d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:02.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bc11d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:02.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:02.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:02.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.128+0000 7f2bbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:02.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.129+0000 7f2bc09d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:02.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.129+0000 7f2bc09d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:02.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.129+0000 7f2bc09d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:02.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bbbfff640 10 monclient(hunting): handle_auth_done global_id 15990 payload 824 2023-10-30T17:55:02.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bbbfff640 1 -- 172.21.15.170:0/1563328916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13a970 msgr2=0x7f2bbc1d7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bbbfff640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13a970 0x7f2bbc1d7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bbbfff640 1 -- 172.21.15.170:0/1563328916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bbc06ceb0 msgr2=0x7f2bbc13a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bbbfff640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bbc06ceb0 0x7f2bbc13a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bbbfff640 10 monclient: _finish_hunting 0 2023-10-30T17:55:02.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bbbfff640 1 monclient: found mon.noname-a 2023-10-30T17:55:02.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:02.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bbbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:02.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bbbfff640 1 -- 172.21.15.170:0/1563328916 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba8009400 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.131+0000 7f2bc11d1640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13a970 0x7f2bbc1d7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T17:55:02.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.133+0000 7f2bbbfff640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2bbc0b0ec0 0x7f2bbc1d6d00 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80098c0 tx=0x7f2ba80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:02.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.133+0000 7f2bb97fa640 10 client.?.objecter ms_handle_connect 0x7f2bbc0b0ec0 2023-10-30T17:55:02.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.133+0000 7f2bb97fa640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:02.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 1 -- 172.21.15.170:0/1563328916 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8032840 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:02.164 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:02.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 10 monclient: dump: 2023-10-30T17:55:02.165 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:02.165 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:02.165 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.165 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.166 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:02.166 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.166 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:02.166 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:02.166 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:02.167 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 10 monclient: _finish_auth 0 2023-10-30T17:55:02.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 10 monclient: _check_auth_tickets 2023-10-30T17:55:02.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:02.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 1 -- 172.21.15.170:0/1563328916 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba8031f00 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:02.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 1 -- 172.21.15.170:0/1563328916 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8032be0 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bc2c34640 5 monclient: authenticate success, global_id 15990 2023-10-30T17:55:02.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bc2c34640 10 monclient: _renew_subs 2023-10-30T17:55:02.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bc2c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:02.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bc2c34640 1 -- 172.21.15.170:0/1563328916 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc0ad910 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bbaffd640 4 set_mon_vals no callback set 2023-10-30T17:55:02.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:02.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:02.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bb97fa640 10 monclient: dump: 2023-10-30T17:55:02.170 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:02.170 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:02.170 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.171 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.171 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:02.171 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.171 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:02.171 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:02.172 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:02.172 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bc2c34640 10 client.15990.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:02.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bc2c34640 10 monclient: _renew_subs 2023-10-30T17:55:02.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bc2c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:02.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bc2c34640 1 -- 172.21.15.170:0/1563328916 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bbc130410 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.137+0000 7f2bc2c34640 1 librados: init done 2023-10-30T17:55:02.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.138+0000 7f2bc2c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:02.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.138+0000 7f2bc2c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:02.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.138+0000 7f2bc2c34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:02.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.138+0000 7f2bc2c34640 1 -- 172.21.15.170:0/1563328916 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bbc13ad80 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.156+0000 7f2bb97fa640 1 -- 172.21.15.170:0/1563328916 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2ba8032d40 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.156+0000 7f2bb97fa640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2ba006c830 0x7f2ba006ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.156+0000 7f2bc09d0640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2ba006c830 0x7f2ba006ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.156+0000 7f2bc09d0640 10 monclient: get_auth_request con 0x7f2ba006c830 auth_method 0 2023-10-30T17:55:02.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.157+0000 7f2bb97fa640 10 client.15990.objecter ms_handle_connect 0x7f2ba006c830 2023-10-30T17:55:02.175 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.157+0000 7f2bc09d0640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2ba006c830 0x7f2ba006ecf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0005880 tx=0x7f2bb0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:02.397 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.396+0000 7f2bb97fa640 1 -- 172.21.15.170:0/1563328916 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(913..913 src has 374..913) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f2ba807a510 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.396+0000 7f2bb97fa640 10 client.15990.objecter ms_dispatch 0x7f2bbc000b90 osd_map(913..913 src has 374..913) v4 2023-10-30T17:55:02.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.396+0000 7f2bb97fa640 3 client.15990.objecter handle_osd_map got epochs [913,913] > 0 2023-10-30T17:55:02.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.396+0000 7f2bb97fa640 3 client.15990.objecter handle_osd_map decoding full epoch 913 2023-10-30T17:55:02.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.396+0000 7f2bb97fa640 20 client.15990.objecter dump_active .. 0 homeless 2023-10-30T17:55:02.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.396+0000 7f2bb97fa640 1 -- 172.21.15.170:0/1563328916 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ba8076850 con 0x7f2bbc0b0ec0 2023-10-30T17:55:02.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.396+0000 7f2bb97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:02.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.396+0000 7f2bb97fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:02.601 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.600+0000 7f2bc2c34640 1 -- 172.21.15.170:0/1563328916 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2bbc139a70 con 0x7f2ba006c830 2023-10-30T17:55:02.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.604+0000 7f2bb97fa640 1 -- 172.21.15.170:0/1563328916 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151540 (secure 0 0 0) 0x7f2bb000f820 con 0x7f2ba006c830 2023-10-30T17:55:02.607 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:55:02.608 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-10-30T17:55:02.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 10 librados: watch_flush enter 2023-10-30T17:55:02.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 10 librados: watch_flush exit 2023-10-30T17:55:02.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 20 client.15990.objecter shutdown clearing up homeless session... 2023-10-30T17:55:02.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 10 client.15990.objecter successfully canceled tick 2023-10-30T17:55:02.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 -- 172.21.15.170:0/1563328916 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2ba006c830 msgr2=0x7f2ba006ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2ba006c830 0x7f2ba006ecf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0005880 tx=0x7f2bb0009040 comp rx=0 tx=0).stop 2023-10-30T17:55:02.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 10 monclient: shutdown 2023-10-30T17:55:02.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:02.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 -- 172.21.15.170:0/1563328916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2bbc0b0ec0 msgr2=0x7f2bbc1d6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2bbc0b0ec0 0x7f2bbc1d6d00 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80098c0 tx=0x7f2ba80046c0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 -- 172.21.15.170:0/1563328916 shutdown_connections 2023-10-30T17:55:02.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2ba006c830 0x7f2ba006ecf0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2bbc13a970 0x7f2bbc1d7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2bbc0b0ec0 0x7f2bbc1d6d00 unknown :-1 s=CLOSED pgs=3363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 --2- 172.21.15.170:0/1563328916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bbc06ceb0 0x7f2bbc13a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 -- 172.21.15.170:0/1563328916 >> 172.21.15.170:0/1563328916 conn(0x7f2bbc0b1d40 msgr2=0x7f2bbc134ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:02.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 -- 172.21.15.170:0/1563328916 shutdown_connections 2023-10-30T17:55:02.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 -- 172.21.15.170:0/1563328916 wait complete. 2023-10-30T17:55:02.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.607+0000 7f2bc2c34640 1 librados: shutdown 2023-10-30T17:55:02.619 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2023-10-30T17:55:01.325774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":568424,"kb_used_data":60776,"kb_used_omap":12,"kb_used_meta":507507,"kb_avail":754406296,"statfs":{"total":773094113280,"available":772512047104,"internally_reserved":0,"allocated":62234624,"data_stored":34698846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":519687736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":375,"apply_latency_ms":375,"commit_latency_ns":375000000,"apply_latency_ns":375000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000948"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":856,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.965106+0000","last_change":"2023-10-30T17:33:37.648571+0000","last_active":"2023-10-30T17:54:56.965106+0000","last_peered":"2023-10-30T17:54:56.965106+0000","last_clean":"2023-10-30T17:54:56.965106+0000","last_became_active":"2023-10-30T17:33:37.643213+0000","last_became_peered":"2023-10-30T17:33:37.643213+0000","last_unstale":"2023-10-30T17:54:56.965106+0000","last_undegraded":"2023-10-30T17:54:56.965106+0000","last_fullsized":"2023-10-30T17:54:56.965106+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:43:30.718487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":853,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.252428+0000","last_change":"2023-10-30T17:33:37.836041+0000","last_active":"2023-10-30T17:54:59.252428+0000","last_peered":"2023-10-30T17:54:59.252428+0000","last_clean":"2023-10-30T17:54:59.252428+0000","last_became_active":"2023-10-30T17:33:37.827806+0000","last_became_peered":"2023-10-30T17:33:37.827806+0000","last_unstale":"2023-10-30T17:54:59.252428+0000","last_undegraded":"2023-10-30T17:54:59.252428+0000","last_fullsized":"2023-10-30T17:54:59.252428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:00.255976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":851,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.737796+0000","last_change":"2023-10-30T17:33:40.825119+0000","last_active":"2023-10-30T17:54:55.737796+0000","last_peered":"2023-10-30T17:54:55.737796+0000","last_clean":"2023-10-30T17:54:55.737796+0000","last_became_active":"2023-10-30T17:33:40.511287+0000","last_became_peered":"2023-10-30T17:33:40.511287+0000","last_unstale":"2023-10-30T17:54:55.737796+0000","last_undegraded":"2023-10-30T17:54:55.737796+0000","last_fullsized":"2023-10-30T17:54:55.737796+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:14:23.464432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":853,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.248681+0000","last_change":"2023-10-30T17:33:38.590694+0000","last_active":"2023-10-30T17:54:59.248681+0000","last_peered":"2023-10-30T17:54:59.248681+0000","last_clean":"2023-10-30T17:54:59.248681+0000","last_became_active":"2023-10-30T17:33:38.581457+0000","last_became_peered":"2023-10-30T17:33:38.581457+0000","last_unstale":"2023-10-30T17:54:59.248681+0000","last_undegraded":"2023-10-30T17:54:59.248681+0000","last_fullsized":"2023-10-30T17:54:59.248681+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:24:17.385325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":856,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.939866+0000","last_change":"2023-10-30T17:33:46.635360+0000","last_active":"2023-10-30T17:54:56.939866+0000","last_peered":"2023-10-30T17:54:56.939866+0000","last_clean":"2023-10-30T17:54:56.939866+0000","last_became_active":"2023-10-30T17:33:46.629442+0000","last_became_peered":"2023-10-30T17:33:46.629442+0000","last_unstale":"2023-10-30T17:54:56.939866+0000","last_undegraded":"2023-10-30T17:54:56.939866+0000","last_fullsized":"2023-10-30T17:54:56.939866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:46:06.082348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":852,"reported_epoch":909,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.139730+0000","last_change":"2023-10-30T17:33:39.141659+0000","last_active":"2023-10-30T17:54:57.139730+0000","last_peered":"2023-10-30T17:54:57.139730+0000","last_clean":"2023-10-30T17:54:57.139730+0000","last_became_active":"2023-10-30T17:33:39.135750+0000","last_became_peered":"2023-10-30T17:33:39.135750+0000","last_unstale":"2023-10-30T17:54:57.139730+0000","last_undegraded":"2023-10-30T17:54:57.139730+0000","last_fullsized":"2023-10-30T17:54:57.139730+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:27.698297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":853,"reported_epoch":909,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.207778+0000","last_change":"2023-10-30T17:33:51.011431+0000","last_active":"2023-10-30T17:54:57.207778+0000","last_peered":"2023-10-30T17:54:57.207778+0000","last_clean":"2023-10-30T17:54:57.207778+0000","last_became_active":"2023-10-30T17:33:51.001524+0000","last_became_peered":"2023-10-30T17:33:51.001524+0000","last_unstale":"2023-10-30T17:54:57.207778+0000","last_undegraded":"2023-10-30T17:54:57.207778+0000","last_fullsized":"2023-10-30T17:54:57.207778+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:41.602832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.027235+0000","last_change":"2023-10-30T17:33:58.879816+0000","last_active":"2023-10-30T17:54:58.027235+0000","last_peered":"2023-10-30T17:54:58.027235+0000","last_clean":"2023-10-30T17:54:58.027235+0000","last_became_active":"2023-10-30T17:33:58.874661+0000","last_became_peered":"2023-10-30T17:33:58.874661+0000","last_unstale":"2023-10-30T17:54:58.027235+0000","last_undegraded":"2023-10-30T17:54:58.027235+0000","last_fullsized":"2023-10-30T17:54:58.027235+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:04:24.038125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":849,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.993781+0000","last_change":"2023-10-30T17:33:39.104029+0000","last_active":"2023-10-30T17:54:58.993781+0000","last_peered":"2023-10-30T17:54:58.993781+0000","last_clean":"2023-10-30T17:54:58.993781+0000","last_became_active":"2023-10-30T17:33:39.086390+0000","last_became_peered":"2023-10-30T17:33:39.086390+0000","last_unstale":"2023-10-30T17:54:58.993781+0000","last_undegraded":"2023-10-30T17:54:58.993781+0000","last_fullsized":"2023-10-30T17:54:58.993781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:52.540858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":853,"reported_epoch":909,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.186394+0000","last_change":"2023-10-30T17:33:40.038786+0000","last_active":"2023-10-30T17:54:57.186394+0000","last_peered":"2023-10-30T17:54:57.186394+0000","last_clean":"2023-10-30T17:54:57.186394+0000","last_became_active":"2023-10-30T17:33:39.965244+0000","last_became_peered":"2023-10-30T17:33:39.965244+0000","last_unstale":"2023-10-30T17:54:57.186394+0000","last_undegraded":"2023-10-30T17:54:57.186394+0000","last_fullsized":"2023-10-30T17:54:57.186394+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:17:21.402055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":857,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.954339+0000","last_change":"2023-10-30T17:33:46.567515+0000","last_active":"2023-10-30T17:54:56.954339+0000","last_peered":"2023-10-30T17:54:56.954339+0000","last_clean":"2023-10-30T17:54:56.954339+0000","last_became_active":"2023-10-30T17:33:46.562080+0000","last_became_peered":"2023-10-30T17:33:46.562080+0000","last_unstale":"2023-10-30T17:54:56.954339+0000","last_undegraded":"2023-10-30T17:54:56.954339+0000","last_fullsized":"2023-10-30T17:54:56.954339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:42:52.518433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.979403+0000","last_change":"2023-10-30T17:33:40.831046+0000","last_active":"2023-10-30T17:54:57.979403+0000","last_peered":"2023-10-30T17:54:57.979403+0000","last_clean":"2023-10-30T17:54:57.979403+0000","last_became_active":"2023-10-30T17:33:40.798989+0000","last_became_peered":"2023-10-30T17:33:40.798989+0000","last_unstale":"2023-10-30T17:54:57.979403+0000","last_undegraded":"2023-10-30T17:54:57.979403+0000","last_fullsized":"2023-10-30T17:54:57.979403+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:35.578143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":853,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.023321+0000","last_change":"2023-10-30T17:32:35.372482+0000","last_active":"2023-10-30T17:54:59.023321+0000","last_peered":"2023-10-30T17:54:59.023321+0000","last_clean":"2023-10-30T17:54:59.023321+0000","last_became_active":"2023-10-30T17:32:35.367487+0000","last_became_peered":"2023-10-30T17:32:35.367487+0000","last_unstale":"2023-10-30T17:54:59.023321+0000","last_undegraded":"2023-10-30T17:54:59.023321+0000","last_fullsized":"2023-10-30T17:54:59.023321+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:45:03.846674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":858,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.920758+0000","last_change":"2023-10-30T17:33:38.291772+0000","last_active":"2023-10-30T17:54:56.920758+0000","last_peered":"2023-10-30T17:54:56.920758+0000","last_clean":"2023-10-30T17:54:56.920758+0000","last_became_active":"2023-10-30T17:33:38.284317+0000","last_became_peered":"2023-10-30T17:33:38.284317+0000","last_unstale":"2023-10-30T17:54:56.920758+0000","last_undegraded":"2023-10-30T17:54:56.920758+0000","last_fullsized":"2023-10-30T17:54:56.920758+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:02.583403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":863,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.945069+0000","last_change":"2023-10-30T17:32:34.589393+0000","last_active":"2023-10-30T17:54:56.945069+0000","last_peered":"2023-10-30T17:54:56.945069+0000","last_clean":"2023-10-30T17:54:56.945069+0000","last_became_active":"2023-10-30T17:32:34.557021+0000","last_became_peered":"2023-10-30T17:32:34.557021+0000","last_unstale":"2023-10-30T17:54:56.945069+0000","last_undegraded":"2023-10-30T17:54:56.945069+0000","last_fullsized":"2023-10-30T17:54:56.945069+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:05.352859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"47'7","reported_seq":863,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.992239+0000","last_change":"2023-10-30T17:33:39.222547+0000","last_active":"2023-10-30T17:54:58.992239+0000","last_peered":"2023-10-30T17:54:58.992239+0000","last_clean":"2023-10-30T17:54:58.992239+0000","last_became_active":"2023-10-30T17:33:39.211129+0000","last_became_peered":"2023-10-30T17:33:39.211129+0000","last_unstale":"2023-10-30T17:54:58.992239+0000","last_undegraded":"2023-10-30T17:54:58.992239+0000","last_fullsized":"2023-10-30T17:54:58.992239+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:49.479747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.008700+0000","last_change":"2023-10-30T17:32:35.362652+0000","last_active":"2023-10-30T17:54:59.008700+0000","last_peered":"2023-10-30T17:54:59.008700+0000","last_clean":"2023-10-30T17:54:59.008700+0000","last_became_active":"2023-10-30T17:32:35.357489+0000","last_became_peered":"2023-10-30T17:32:35.357489+0000","last_unstale":"2023-10-30T17:54:59.008700+0000","last_undegraded":"2023-10-30T17:54:59.008700+0000","last_fullsized":"2023-10-30T17:54:59.008700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:21:36.073054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":855,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.609324+0000","last_change":"2023-10-30T17:33:40.644802+0000","last_active":"2023-10-30T17:54:54.609324+0000","last_peered":"2023-10-30T17:54:54.609324+0000","last_clean":"2023-10-30T17:54:54.609324+0000","last_became_active":"2023-10-30T17:33:40.584323+0000","last_became_peered":"2023-10-30T17:33:40.584323+0000","last_unstale":"2023-10-30T17:54:54.609324+0000","last_undegraded":"2023-10-30T17:54:54.609324+0000","last_fullsized":"2023-10-30T17:54:54.609324+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:41:03.792929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":860,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.906532+0000","last_change":"2023-10-30T17:32:34.658442+0000","last_active":"2023-10-30T17:54:55.906532+0000","last_peered":"2023-10-30T17:54:55.906532+0000","last_clean":"2023-10-30T17:54:55.906532+0000","last_became_active":"2023-10-30T17:32:34.631494+0000","last_became_peered":"2023-10-30T17:32:34.631494+0000","last_unstale":"2023-10-30T17:54:55.906532+0000","last_undegraded":"2023-10-30T17:54:55.906532+0000","last_fullsized":"2023-10-30T17:54:55.906532+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:36:36.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":858,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.956509+0000","last_change":"2023-10-30T17:33:46.593988+0000","last_active":"2023-10-30T17:54:56.956509+0000","last_peered":"2023-10-30T17:54:56.956509+0000","last_clean":"2023-10-30T17:54:56.956509+0000","last_became_active":"2023-10-30T17:33:46.588555+0000","last_became_peered":"2023-10-30T17:33:46.588555+0000","last_unstale":"2023-10-30T17:54:56.956509+0000","last_undegraded":"2023-10-30T17:54:56.956509+0000","last_fullsized":"2023-10-30T17:54:56.956509+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:57.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.998017+0000","last_change":"2023-10-30T17:32:35.388556+0000","last_active":"2023-10-30T17:54:58.998017+0000","last_peered":"2023-10-30T17:54:58.998017+0000","last_clean":"2023-10-30T17:54:58.998017+0000","last_became_active":"2023-10-30T17:32:35.383414+0000","last_became_peered":"2023-10-30T17:32:35.383414+0000","last_unstale":"2023-10-30T17:54:58.998017+0000","last_undegraded":"2023-10-30T17:54:58.998017+0000","last_fullsized":"2023-10-30T17:54:58.998017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:05:46.189212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":871,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.595206+0000","last_change":"2023-10-30T17:33:38.185755+0000","last_active":"2023-10-30T17:54:54.595206+0000","last_peered":"2023-10-30T17:54:54.595206+0000","last_clean":"2023-10-30T17:54:54.595206+0000","last_became_active":"2023-10-30T17:33:38.176989+0000","last_became_peered":"2023-10-30T17:33:38.176989+0000","last_unstale":"2023-10-30T17:54:54.595206+0000","last_undegraded":"2023-10-30T17:54:54.595206+0000","last_fullsized":"2023-10-30T17:54:54.595206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:25:37.689974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":878,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.580413+0000","last_change":"2023-10-30T17:32:34.318569+0000","last_active":"2023-10-30T17:54:54.580413+0000","last_peered":"2023-10-30T17:54:54.580413+0000","last_clean":"2023-10-30T17:54:54.580413+0000","last_became_active":"2023-10-30T17:32:34.287031+0000","last_became_peered":"2023-10-30T17:32:34.287031+0000","last_unstale":"2023-10-30T17:54:54.580413+0000","last_undegraded":"2023-10-30T17:54:54.580413+0000","last_fullsized":"2023-10-30T17:54:54.580413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:55:59.080654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":870,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.610146+0000","last_change":"2023-10-30T17:33:41.123654+0000","last_active":"2023-10-30T17:54:54.610146+0000","last_peered":"2023-10-30T17:54:54.610146+0000","last_clean":"2023-10-30T17:54:54.610146+0000","last_became_active":"2023-10-30T17:33:41.116885+0000","last_became_peered":"2023-10-30T17:33:41.116885+0000","last_unstale":"2023-10-30T17:54:54.610146+0000","last_undegraded":"2023-10-30T17:54:54.610146+0000","last_fullsized":"2023-10-30T17:54:54.610146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:51:37.700398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.996395+0000","last_change":"2023-10-30T17:32:35.307241+0000","last_active":"2023-10-30T17:54:58.996395+0000","last_peered":"2023-10-30T17:54:58.996395+0000","last_clean":"2023-10-30T17:54:58.996395+0000","last_became_active":"2023-10-30T17:32:35.277163+0000","last_became_peered":"2023-10-30T17:32:35.277163+0000","last_unstale":"2023-10-30T17:54:58.996395+0000","last_undegraded":"2023-10-30T17:54:58.996395+0000","last_fullsized":"2023-10-30T17:54:58.996395+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:02.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":848,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.025231+0000","last_change":"2023-10-30T17:33:43.745646+0000","last_active":"2023-10-30T17:54:59.025231+0000","last_peered":"2023-10-30T17:54:59.025231+0000","last_clean":"2023-10-30T17:54:59.025231+0000","last_became_active":"2023-10-30T17:33:43.736021+0000","last_became_peered":"2023-10-30T17:33:43.736021+0000","last_unstale":"2023-10-30T17:54:59.025231+0000","last_undegraded":"2023-10-30T17:54:59.025231+0000","last_fullsized":"2023-10-30T17:54:59.025231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:29.825853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":854,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.021371+0000","last_change":"2023-10-30T17:32:35.628758+0000","last_active":"2023-10-30T17:54:59.021371+0000","last_peered":"2023-10-30T17:54:59.021371+0000","last_clean":"2023-10-30T17:54:59.021371+0000","last_became_active":"2023-10-30T17:32:35.623395+0000","last_became_peered":"2023-10-30T17:32:35.623395+0000","last_unstale":"2023-10-30T17:54:59.021371+0000","last_undegraded":"2023-10-30T17:54:59.021371+0000","last_fullsized":"2023-10-30T17:54:59.021371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:47:08.984576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":966,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.254775+0000","last_change":"2023-10-30T17:32:28.719757+0000","last_active":"2023-10-30T17:54:59.254775+0000","last_peered":"2023-10-30T17:54:59.254775+0000","last_clean":"2023-10-30T17:54:59.254775+0000","last_became_active":"2023-10-30T17:32:28.686958+0000","last_became_peered":"2023-10-30T17:32:28.686958+0000","last_unstale":"2023-10-30T17:54:59.254775+0000","last_undegraded":"2023-10-30T17:54:59.254775+0000","last_fullsized":"2023-10-30T17:54:59.254775+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:25.778908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:25.778908+0000","last_clean_scrub_stamp":"2023-10-30T17:32:25.778908+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:41:21.329138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":855,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.630478+0000","last_change":"2023-10-30T17:33:40.638676+0000","last_active":"2023-10-30T17:54:54.630478+0000","last_peered":"2023-10-30T17:54:54.630478+0000","last_clean":"2023-10-30T17:54:54.630478+0000","last_became_active":"2023-10-30T17:33:40.602436+0000","last_became_peered":"2023-10-30T17:33:40.602436+0000","last_unstale":"2023-10-30T17:54:54.630478+0000","last_undegraded":"2023-10-30T17:54:54.630478+0000","last_fullsized":"2023-10-30T17:54:54.630478+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:41.628722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":849,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.013211+0000","last_change":"2023-10-30T17:33:38.623594+0000","last_active":"2023-10-30T17:54:59.013211+0000","last_peered":"2023-10-30T17:54:59.013211+0000","last_clean":"2023-10-30T17:54:59.013211+0000","last_became_active":"2023-10-30T17:33:38.616054+0000","last_became_peered":"2023-10-30T17:33:38.616054+0000","last_unstale":"2023-10-30T17:54:59.013211+0000","last_undegraded":"2023-10-30T17:54:59.013211+0000","last_fullsized":"2023-10-30T17:54:59.013211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:05:44.520215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":854,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.272308+0000","last_change":"2023-10-30T17:33:40.742636+0000","last_active":"2023-10-30T17:54:59.272308+0000","last_peered":"2023-10-30T17:54:59.272308+0000","last_clean":"2023-10-30T17:54:59.272308+0000","last_became_active":"2023-10-30T17:33:40.735511+0000","last_became_peered":"2023-10-30T17:33:40.735511+0000","last_unstale":"2023-10-30T17:54:59.272308+0000","last_undegraded":"2023-10-30T17:54:59.272308+0000","last_fullsized":"2023-10-30T17:54:59.272308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:30.315904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":854,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.905041+0000","last_change":"2023-10-30T17:33:39.276657+0000","last_active":"2023-10-30T17:54:55.905041+0000","last_peered":"2023-10-30T17:54:55.905041+0000","last_clean":"2023-10-30T17:54:55.905041+0000","last_became_active":"2023-10-30T17:33:39.228561+0000","last_became_peered":"2023-10-30T17:33:39.228561+0000","last_unstale":"2023-10-30T17:54:55.905041+0000","last_undegraded":"2023-10-30T17:54:55.905041+0000","last_fullsized":"2023-10-30T17:54:55.905041+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:01.276873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":870,"reported_epoch":907,"state":"active+clean","last_fresh":"2023-10-30T17:54:54.597303+0000","last_change":"2023-10-30T17:33:41.611205+0000","last_active":"2023-10-30T17:54:54.597303+0000","last_peered":"2023-10-30T17:54:54.597303+0000","last_clean":"2023-10-30T17:54:54.597303+0000","last_became_active":"2023-10-30T17:33:41.605208+0000","last_became_peered":"2023-10-30T17:33:41.605208+0000","last_unstale":"2023-10-30T17:54:54.597303+0000","last_undegraded":"2023-10-30T17:54:54.597303+0000","last_fullsized":"2023-10-30T17:54:54.597303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:09.587584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:57.980989+0000","last_change":"2023-10-30T17:33:44.441890+0000","last_active":"2023-10-30T17:54:57.980989+0000","last_peered":"2023-10-30T17:54:57.980989+0000","last_clean":"2023-10-30T17:54:57.980989+0000","last_became_active":"2023-10-30T17:33:44.434104+0000","last_became_peered":"2023-10-30T17:33:44.434104+0000","last_unstale":"2023-10-30T17:54:57.980989+0000","last_undegraded":"2023-10-30T17:54:57.980989+0000","last_fullsized":"2023-10-30T17:54:57.980989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:11.365044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":855,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:55.899091+0000","last_change":"2023-10-30T17:33:43.330759+0000","last_active":"2023-10-30T17:54:55.899091+0000","last_peered":"2023-10-30T17:54:55.899091+0000","last_clean":"2023-10-30T17:54:55.899091+0000","last_became_active":"2023-10-30T17:33:43.322860+0000","last_became_peered":"2023-10-30T17:33:43.322860+0000","last_unstale":"2023-10-30T17:54:55.899091+0000","last_undegraded":"2023-10-30T17:54:55.899091+0000","last_fullsized":"2023-10-30T17:54:55.899091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:56:01.616064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":868,"reported_epoch":910,"state":"active+clean","last_fresh":"2023-10-30T17:54:58.020934+0000","last_change":"2023-10-30T17:33:53.140054+0000","last_active":"2023-10-30T17:54:58.020934+0000","last_peered":"2023-10-30T17:54:58.020934+0000","last_clean":"2023-10-30T17:54:58.020934+0000","last_became_active":"2023-10-30T17:33:53.130615+0000","last_became_peered":"2023-10-30T17:33:53.130615+0000","last_unstale":"2023-10-30T17:54:58.020934+0000","last_undegraded":"2023-10-30T17:54:58.020934+0000","last_fullsized":"2023-10-30T17:54:58.020934+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:11.025225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":854,"reported_epoch":912,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.244856+0000","last_change":"2023-10-30T17:33:37.571189+0000","last_active":"2023-10-30T17:54:59.244856+0000","last_peered":"2023-10-30T17:54:59.244856+0000","last_clean":"2023-10-30T17:54:59.244856+0000","last_became_active":"2023-10-30T17:33:37.540749+0000","last_became_peered":"2023-10-30T17:33:37.540749+0000","last_unstale":"2023-10-30T17:54:59.244856+0000","last_undegraded":"2023-10-30T17:54:59.244856+0000","last_fullsized":"2023-10-30T17:54:59.244856+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:42.432458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":848,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.016385+0000","last_change":"2023-10-30T17:33:58.392321+0000","last_active":"2023-10-30T17:54:59.016385+0000","last_peered":"2023-10-30T17:54:59.016385+0000","last_clean":"2023-10-30T17:54:59.016385+0000","last_became_active":"2023-10-30T17:33:58.383320+0000","last_became_peered":"2023-10-30T17:33:58.383320+0000","last_unstale":"2023-10-30T17:54:59.016385+0000","last_undegraded":"2023-10-30T17:54:59.016385+0000","last_fullsized":"2023-10-30T17:54:59.016385+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:05.698087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":855,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.949358+0000","last_change":"2023-10-30T17:33:38.595237+0000","last_active":"2023-10-30T17:54:56.949358+0000","last_peered":"2023-10-30T17:54:56.949358+0000","last_clean":"2023-10-30T17:54:56.949358+0000","last_became_active":"2023-10-30T17:33:38.589479+0000","last_became_peered":"2023-10-30T17:33:38.589479+0000","last_unstale":"2023-10-30T17:54:56.949358+0000","last_undegraded":"2023-10-30T17:54:56.949358+0000","last_fullsized":"2023-10-30T17:54:56.949358+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:17:44.312113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":857,"reported_epoch":908,"state":"active+clean","last_fresh":"2023-10-30T17:54:56.959591+0000","last_change":"2023-10-30T17:33:40.505225+0000","last_active":"2023-10-30T17:54:56.959591+0000","last_peered":"2023-10-30T17:54:56.959591+0000","last_clean":"2023-10-30T17:54:56.959591+0000","last_became_active":"2023-10-30T17:33:40.476315+0000","last_became_peered":"2023-10-30T17:33:40.476315+0000","last_unstale":"2023-10-30T17:54:56.959591+0000","last_undegraded":"2023-10-30T17:54:56.959591+0000","last_fullsized":"2023-10-30T17:54:56.959591+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:57:02.608925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":848,"reported_epoch":911,"state":"active+clean","last_fresh":"2023-10-30T17:54:59.006691+0000","last_change":"2023-10-30T17:33:38.803571+0000","last_active":"2023-10-30T17:54:59.006691+0000","last_peered":"2023-10-30T17:54:59.006691+0000","last_clean":"2023-10-30T17:54:59.006691+0000","last_became_active":"2023-10-30T17:33:38.795801+0000","last_became_peered":"2023-10-30T17:33:38.795801+0000","last_unstale":"2023-10-30T17:54:59.006691+0000","last_undegraded":"2023-10-30T17:54:59.006691+0000","last_fullsized":"2023-10-30T17:54:59.006691+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:40:52.889892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":17,"seq":73014444302,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60904,"kb_used_data":7512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310936,"statfs":{"total":96636764160,"available":96574398464,"internally_reserved":0,"allocated":7692288,"data_stored":4265037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":7.8239999999999998,"5min":14.673999999999999,"15min":20.202999999999999},"min":{"1min":4.2460000000000004,"5min":4.2460000000000004,"15min":4.2460000000000004},"max":{"1min":15.343,"5min":217.97399999999999,"15min":296.49400000000003},"last":6.1900000000000004},{"interface":"front","average":{"1min":13.112,"5min":16.077000000000002,"15min":21.355},"min":{"1min":7.6120000000000001,"5min":5.6189999999999998,"15min":4.6299999999999999},"max":{"1min":31.678999999999998,"5min":134.833,"15min":276.49799999999999},"last":8.9879999999999995}]},{"osd":2,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":16.689,"5min":19.114999999999998,"15min":26.032},"min":{"1min":6.319,"5min":5.915,"15min":5.915},"max":{"1min":101.64100000000001,"5min":218.22399999999999,"15min":591.02499999999998},"last":11.734},{"interface":"front","average":{"1min":19.178999999999998,"5min":20.963000000000001,"15min":25.277000000000001},"min":{"1min":9.0850000000000009,"5min":7.6520000000000001,"15min":6.4930000000000003},"max":{"1min":103.506,"5min":228.36500000000001,"15min":259.52300000000002},"last":10.427}]},{"osd":3,"last update":"Mon Oct 30 17:53:58 2023","interfaces":[{"interface":"back","average":{"1min":23.952999999999999,"5min":78.492000000000004,"15min":43.039000000000001},"min":{"1min":6.9329999999999998,"5min":6.6879999999999997,"15min":5.8099999999999996},"max":{"1min":282.685,"5min":5036.1790000000001,"15min":5036.1790000000001},"last":9.7240000000000002},{"interface":"front","average":{"1min":24.942,"5min":81.421000000000006,"15min":47.087000000000003},"min":{"1min":8.4399999999999995,"5min":7.2770000000000001,"15min":6.4909999999999997},"max":{"1min":190.00899999999999,"5min":5057.1229999999996,"15min":5057.1229999999996},"last":13.465999999999999}]},{"osd":4,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":24.963000000000001,"5min":22.420999999999999,"15min":22.024000000000001},"min":{"1min":7.3259999999999996,"5min":7.3259999999999996,"15min":6.3689999999999998},"max":{"1min":159.94399999999999,"5min":204.91,"15min":281.16399999999999},"last":16.094000000000001},{"interface":"front","average":{"1min":29.393999999999998,"5min":26.954000000000001,"15min":27.579000000000001},"min":{"1min":9.0530000000000008,"5min":9.0530000000000008,"15min":5.8879999999999999},"max":{"1min":190.24199999999999,"5min":218.85300000000001,"15min":297.57299999999998},"last":14.125}]},{"osd":5,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":19.146999999999998,"5min":25.224,"15min":28.202999999999999},"min":{"1min":8.6660000000000004,"5min":8.4960000000000004,"15min":7.9370000000000003},"max":{"1min":44.509,"5min":260.81099999999998,"15min":260.81099999999998},"last":18.850000000000001},{"interface":"front","average":{"1min":15.294,"5min":22.741,"15min":25.870000000000001},"min":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.1189999999999998},"max":{"1min":32.457999999999998,"5min":236.83500000000001,"15min":236.83500000000001},"last":16.760999999999999}]},{"osd":6,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":19.766999999999999,"5min":22.66,"15min":27.329000000000001},"min":{"1min":10.051,"5min":9.968,"15min":8.4019999999999992},"max":{"1min":37.848999999999997,"5min":125.158,"15min":292.47500000000002},"last":17.888000000000002},{"interface":"front","average":{"1min":20.893000000000001,"5min":26.652000000000001,"15min":28.780000000000001},"min":{"1min":11.971,"5min":6.8449999999999998,"15min":6.8449999999999998},"max":{"1min":62.518000000000001,"5min":189.01599999999999,"15min":273.64999999999998},"last":15.422000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":22.042999999999999,"5min":25.934999999999999,"15min":29.561},"min":{"1min":12.084,"5min":10.622999999999999,"15min":9.157},"max":{"1min":60.759,"5min":204.12200000000001,"15min":262.74200000000002},"last":12.778},{"interface":"front","average":{"1min":26.411999999999999,"5min":31.727,"15min":33.009999999999998},"min":{"1min":13.321,"5min":12.188000000000001,"15min":10.122999999999999},"max":{"1min":100.953,"5min":219.833,"15min":264.81},"last":19.789000000000001}]}]},{"osd":3,"up_from":16,"seq":68719476998,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83092,"kb_used_data":7492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288748,"statfs":{"total":96636764160,"available":96551677952,"internally_reserved":0,"allocated":7671808,"data_stored":4216833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":9.1400000000000006,"5min":16.59,"15min":16.803999999999998},"min":{"1min":4.734,"5min":4.3869999999999996,"15min":4.3869999999999996},"max":{"1min":19.263000000000002,"5min":203.85499999999999,"15min":271.12799999999999},"last":9.2040000000000006},{"interface":"front","average":{"1min":12.346,"5min":17.565999999999999,"15min":18.774000000000001},"min":{"1min":5.5490000000000004,"5min":4.8449999999999998,"15min":4.367},"max":{"1min":36.253,"5min":176.83500000000001,"15min":364.05399999999997},"last":11.433999999999999}]},{"osd":1,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":19.100000000000001,"5min":22.856999999999999,"15min":24.716000000000001},"min":{"1min":7.3470000000000004,"5min":6.6589999999999998,"15min":5.5380000000000003},"max":{"1min":74.484999999999999,"5min":178.04300000000001,"15min":314.19099999999997},"last":19.881},{"interface":"front","average":{"1min":24.044,"5min":23.404,"15min":23.681000000000001},"min":{"1min":6.5609999999999999,"5min":5.9290000000000003,"15min":5.609},"max":{"1min":159.42099999999999,"5min":210.381,"15min":240.64500000000001},"last":13.476000000000001}]},{"osd":2,"last update":"Mon Oct 30 17:54:19 2023","interfaces":[{"interface":"back","average":{"1min":34.308999999999997,"5min":27.562000000000001,"15min":24.027999999999999},"min":{"1min":7.2969999999999997,"5min":7.2489999999999997,"15min":5.234},"max":{"1min":251.941,"5min":251.941,"15min":397.50099999999998},"last":17.768999999999998},{"interface":"front","average":{"1min":19.899000000000001,"5min":22.161999999999999,"15min":25.760999999999999},"min":{"1min":7.1539999999999999,"5min":6.54,"15min":6.54},"max":{"1min":85.495000000000005,"5min":147.50399999999999,"15min":219.65000000000001},"last":10.699}]},{"osd":4,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":22.57,"5min":33.298000000000002,"15min":27.831},"min":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":5.8140000000000001},"max":{"1min":164.642,"5min":318.92200000000003,"15min":400.13999999999999},"last":12.052},{"interface":"front","average":{"1min":17.919,"5min":25.715,"15min":22.006},"min":{"1min":8.3650000000000002,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":83.346999999999994,"5min":268.01600000000002,"15min":325.846},"last":18.446999999999999}]},{"osd":5,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":20.472000000000001,"5min":21.146999999999998,"15min":23.655000000000001},"min":{"1min":9.2029999999999994,"5min":7.7569999999999997,"15min":7.7569999999999997},"max":{"1min":84.305000000000007,"5min":149.61600000000001,"15min":221.821},"last":16.879000000000001},{"interface":"front","average":{"1min":30.245000000000001,"5min":27.282,"15min":28.794},"min":{"1min":9.75,"5min":7.2009999999999996,"15min":7.2009999999999996},"max":{"1min":225.15299999999999,"5min":225.15299999999999,"15min":372.13999999999999},"last":15.724}]},{"osd":6,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":32.417999999999999,"5min":30.308,"15min":28.475000000000001},"min":{"1min":10.678000000000001,"5min":7.9589999999999996,"15min":7.7439999999999998},"max":{"1min":272.536,"5min":272.536,"15min":360.26999999999998},"last":58.084000000000003},{"interface":"front","average":{"1min":23.282,"5min":23.863,"15min":25.344000000000001},"min":{"1min":11.023999999999999,"5min":9.3040000000000003,"15min":7.524},"max":{"1min":113.96599999999999,"5min":162.61799999999999,"15min":336.68700000000001},"last":97.793000000000006}]},{"osd":7,"last update":"Mon Oct 30 17:54:06 2023","interfaces":[{"interface":"back","average":{"1min":27.902999999999999,"5min":31.673999999999999,"15min":29.501000000000001},"min":{"1min":12.977,"5min":9.8399999999999999,"15min":9.7059999999999995},"max":{"1min":178.43100000000001,"5min":196.05699999999999,"15min":281.15699999999998},"last":55.932000000000002},{"interface":"front","average":{"1min":31.780999999999999,"5min":30.504999999999999,"15min":29.385999999999999},"min":{"1min":11.692,"5min":10.587999999999999,"15min":10.528},"max":{"1min":177.44900000000001,"5min":182.07499999999999,"15min":233.489},"last":31.356999999999999}]}]},{"osd":1,"up_from":16,"seq":68719477009,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61300,"kb_used_data":7904,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310540,"statfs":{"total":96636764160,"available":96573992960,"internally_reserved":0,"allocated":8093696,"data_stored":4628450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":10.568,"5min":13.151999999999999,"15min":17.620999999999999},"min":{"1min":5.1379999999999999,"5min":4.8099999999999996,"15min":3.8279999999999998},"max":{"1min":26.812000000000001,"5min":162.364,"15min":313.69400000000002},"last":11.885999999999999},{"interface":"front","average":{"1min":13.189,"5min":13.273999999999999,"15min":17.713000000000001},"min":{"1min":8.4429999999999996,"5min":5.4560000000000004,"15min":4.9089999999999998},"max":{"1min":21.195,"5min":59.277000000000001,"15min":209.62799999999999},"last":8.4979999999999993}]},{"osd":2,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":17.059999999999999,"5min":16.346,"15min":21.375},"min":{"1min":8.5259999999999998,"5min":6.2190000000000003,"15min":6.0279999999999996},"max":{"1min":84.930999999999997,"5min":95.995000000000005,"15min":267.83499999999998},"last":14.233000000000001},{"interface":"front","average":{"1min":19.794,"5min":18.084,"15min":20.032},"min":{"1min":11.779999999999999,"5min":7.0270000000000001,"15min":5.8550000000000004},"max":{"1min":89.875,"5min":89.875,"15min":178.44},"last":26.521000000000001}]},{"osd":3,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":10.504,"5min":246.75800000000001,"15min":96.697000000000003},"min":{"1min":6.1180000000000003,"5min":6.1050000000000004,"15min":5.2990000000000004},"max":{"1min":31.059000000000001,"5min":8788.2559999999994,"15min":8788.2559999999994},"last":8.9199999999999999},{"interface":"front","average":{"1min":17.736999999999998,"5min":247.22,"15min":98.808000000000007},"min":{"1min":11.208,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":29.475999999999999,"5min":8677.7369999999992,"15min":8677.7369999999992},"last":29.404}]},{"osd":4,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":15.772,"5min":17.181999999999999,"15min":23.558},"min":{"1min":7.7359999999999998,"5min":6.8220000000000001,"15min":6.8220000000000001},"max":{"1min":91.501999999999995,"5min":91.501999999999995,"15min":235.125},"last":9.5410000000000004},{"interface":"front","average":{"1min":20.797999999999998,"5min":21.347000000000001,"15min":23.934000000000001},"min":{"1min":10.278,"5min":7.673,"15min":7.1829999999999998},"max":{"1min":92.772000000000006,"5min":92.772000000000006,"15min":334.19900000000001},"last":13.321}]},{"osd":5,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":15.865,"5min":19.510000000000002,"15min":27.055},"min":{"1min":9.0389999999999997,"5min":9.0389999999999997,"15min":7.6360000000000001},"max":{"1min":43.468000000000004,"5min":89.837000000000003,"15min":220.65799999999999},"last":28.439},{"interface":"front","average":{"1min":21.504999999999999,"5min":21.619,"15min":31.059999999999999},"min":{"1min":14.56,"5min":8.0869999999999997,"15min":8.0869999999999997},"max":{"1min":50.332000000000001,"5min":134.42599999999999,"15min":262.61799999999999},"last":32.908000000000001}]},{"osd":6,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":10.253,"5min":15.454000000000001,"15min":24.402000000000001},"min":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":6.8769999999999998},"max":{"1min":20.927,"5min":200.369,"15min":249.25299999999999},"last":16.367000000000001},{"interface":"front","average":{"1min":17.808,"5min":20.117999999999999,"15min":21.41},"min":{"1min":10.965,"5min":8.8610000000000007,"15min":6.2880000000000003},"max":{"1min":28.323,"5min":201.756,"15min":259.03899999999999},"last":31.189}]},{"osd":7,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":17.879000000000001,"5min":21.974,"15min":27.113},"min":{"1min":11.935,"5min":9.8200000000000003,"15min":9.4269999999999996},"max":{"1min":29.427,"5min":100.73999999999999,"15min":200.62100000000001},"last":25.783000000000001},{"interface":"front","average":{"1min":21.350999999999999,"5min":26.129999999999999,"15min":30.661000000000001},"min":{"1min":16.481000000000002,"5min":10.682,"15min":10.115},"max":{"1min":36.606999999999999,"5min":203.929,"15min":205.61799999999999},"last":32.268000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509712,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60844,"kb_used_data":7460,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310996,"statfs":{"total":96636764160,"available":96574459904,"internally_reserved":0,"allocated":7639040,"data_stored":4202947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":14.316000000000001,"5min":13.782,"15min":18.59},"min":{"1min":5.3460000000000001,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":106.839,"5min":106.839,"15min":334.36900000000003},"last":6.2960000000000003},{"interface":"front","average":{"1min":18.443000000000001,"5min":17.509,"15min":21.878},"min":{"1min":7.7990000000000004,"5min":7.1319999999999997,"15min":6.7489999999999997},"max":{"1min":111.07599999999999,"5min":111.07599999999999,"15min":283.46899999999999},"last":11.449}]},{"osd":1,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":25.234000000000002,"5min":23.196999999999999,"15min":24.32},"min":{"1min":9.1259999999999994,"5min":6.1539999999999999,"15min":4.7830000000000004},"max":{"1min":122.607,"5min":194.499,"15min":273.93599999999998},"last":12.382999999999999},{"interface":"front","average":{"1min":18.93,"5min":14.327999999999999,"15min":15.942},"min":{"1min":5.0069999999999997,"5min":4.9020000000000001,"15min":4.1890000000000001},"max":{"1min":109.767,"5min":109.767,"15min":152.87299999999999},"last":7.9729999999999999}]},{"osd":3,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":12.736000000000001,"5min":219.62799999999999,"15min":85.793999999999997},"min":{"1min":5.6429999999999998,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":62.838000000000001,"5min":8749.8639999999996,"15min":8749.8639999999996},"last":108.666},{"interface":"front","average":{"1min":19.050999999999998,"5min":225.965,"15min":91.936000000000007},"min":{"1min":9.2469999999999999,"5min":7.7859999999999996,"15min":6.6799999999999997},"max":{"1min":75.923000000000002,"5min":8752.7450000000008,"15min":8752.7450000000008},"last":109.779}]},{"osd":4,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":24.823,"5min":22.719000000000001,"15min":28.533999999999999},"min":{"1min":8.8420000000000005,"5min":8.5990000000000002,"15min":8.5990000000000002},"max":{"1min":115.70999999999999,"5min":124.502,"15min":301.34399999999999},"last":14.112},{"interface":"front","average":{"1min":27.699999999999999,"5min":27.585000000000001,"15min":32.787999999999997},"min":{"1min":11.974,"5min":11.974,"15min":11.824},"max":{"1min":117.285,"5min":134.44800000000001,"15min":300.68900000000002},"last":15.726000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":21.994,"5min":19.698,"15min":29.402999999999999},"min":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":124.914,"5min":198.46700000000001,"15min":318.01900000000001},"last":25.18},{"interface":"front","average":{"1min":21.603999999999999,"5min":18.942,"15min":24.146000000000001},"min":{"1min":10.16,"5min":7.5960000000000001,"15min":5.8920000000000003},"max":{"1min":122.861,"5min":122.861,"15min":325.87099999999998},"last":107.361}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":19.571000000000002,"5min":18.742000000000001,"15min":20.893000000000001},"min":{"1min":8.9290000000000003,"5min":8.7420000000000009,"15min":6.3979999999999997},"max":{"1min":70.927999999999997,"5min":70.927999999999997,"15min":328.983},"last":8.9909999999999997},{"interface":"front","average":{"1min":25.178000000000001,"5min":26.393999999999998,"15min":30.449000000000002},"min":{"1min":11.391,"5min":11.391,"15min":10.986000000000001},"max":{"1min":124.36799999999999,"5min":133.02199999999999,"15min":306.81799999999998},"last":17.527000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":17.791,"5min":16.997,"15min":22.556000000000001},"min":{"1min":7.8630000000000004,"5min":7.2549999999999999,"15min":6.9669999999999996},"max":{"1min":127.804,"5min":127.804,"15min":307.69499999999999},"last":9.9990000000000006},{"interface":"front","average":{"1min":21.841999999999999,"5min":22.962,"15min":28.602},"min":{"1min":10.927,"5min":8.9550000000000001,"15min":8.5700000000000003},"max":{"1min":129.26599999999999,"5min":129.26599999999999,"15min":317.13400000000001},"last":10.691000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83092,"kb_used_data":7492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288748,"statfs":{"total":96636764160,"available":96551677952,"internally_reserved":0,"allocated":7671808,"data_stored":4240466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77396427},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":18.553000000000001,"5min":17.256,"15min":18.238},"min":{"1min":4.6230000000000002,"5min":4.6230000000000002,"15min":4.1580000000000004},"max":{"1min":172.93199999999999,"5min":179.608,"15min":259.12200000000001},"last":5.548},{"interface":"front","average":{"1min":19.984999999999999,"5min":18.443999999999999,"15min":18.128},"min":{"1min":4.5999999999999996,"5min":4.5999999999999996,"15min":4.3959999999999999},"max":{"1min":176.59399999999999,"5min":176.59399999999999,"15min":258.25799999999998},"last":7.1600000000000001}]},{"osd":1,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":13.958,"5min":15.481,"15min":18.420999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.2939999999999996},"max":{"1min":50.183,"5min":83.355000000000004,"15min":249.69900000000001},"last":16.523},{"interface":"front","average":{"1min":16.277000000000001,"5min":19.504000000000001,"15min":24.225999999999999},"min":{"1min":8.0890000000000004,"5min":8.0640000000000001,"15min":5.0439999999999996},"max":{"1min":67.171000000000006,"5min":199.15299999999999,"15min":334.67599999999999},"last":15.115}]},{"osd":2,"last update":"Mon Oct 30 17:54:03 2023","interfaces":[{"interface":"back","average":{"1min":18.440999999999999,"5min":18.277999999999999,"15min":22.067},"min":{"1min":7.282,"5min":6.3410000000000002,"15min":6.3140000000000001},"max":{"1min":107.964,"5min":128.202,"15min":403.90899999999999},"last":17.457000000000001},{"interface":"front","average":{"1min":33.597000000000001,"5min":25.751999999999999,"15min":27.367999999999999},"min":{"1min":9.8949999999999996,"5min":8.4559999999999995,"15min":7.6829999999999998},"max":{"1min":276.67700000000002,"5min":276.67700000000002,"15min":354.34800000000001},"last":25.731000000000002}]},{"osd":3,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":19.672000000000001,"5min":165.75800000000001,"15min":72.480000000000004},"min":{"1min":8.8859999999999992,"5min":8.8859999999999992,"15min":6.226},"max":{"1min":71.772999999999996,"5min":8740.9120000000003,"15min":8740.9120000000003},"last":23.808},{"interface":"front","average":{"1min":16.125,"5min":162.107,"15min":70.201999999999998},"min":{"1min":7.444,"5min":7.3840000000000003,"15min":6.1120000000000001},"max":{"1min":50.947000000000003,"5min":8827.2800000000007,"15min":8827.2800000000007},"last":21.263000000000002}]},{"osd":4,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":17.946999999999999,"5min":23.007999999999999,"15min":24.891999999999999},"min":{"1min":6.758,"5min":6.758,"15min":6.758},"max":{"1min":73.082999999999998,"5min":193.517,"15min":265.78300000000002},"last":20.126999999999999},{"interface":"front","average":{"1min":25.338000000000001,"5min":28.114999999999998,"15min":26.849},"min":{"1min":11.759,"5min":10.228999999999999,"15min":8.5839999999999996},"max":{"1min":121.331,"5min":204.27799999999999,"15min":297.99900000000002},"last":26.582000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:54:03 2023","interfaces":[{"interface":"back","average":{"1min":21.972000000000001,"5min":25.536000000000001,"15min":30.370000000000001},"min":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":7.7729999999999997},"max":{"1min":96.088999999999999,"5min":235.08500000000001,"15min":396.77300000000002},"last":15.843999999999999},{"interface":"front","average":{"1min":20.291,"5min":21.542000000000002,"15min":25.526},"min":{"1min":8.0500000000000007,"5min":6.5010000000000003,"15min":5.5010000000000003},"max":{"1min":103.301,"5min":266.30200000000002,"15min":333.88200000000001},"last":21.800999999999998}]},{"osd":6,"last update":"Mon Oct 30 17:54:03 2023","interfaces":[{"interface":"back","average":{"1min":22.289000000000001,"5min":23.263000000000002,"15min":25.934999999999999},"min":{"1min":9.2609999999999992,"5min":9.1809999999999992,"15min":6.9400000000000004},"max":{"1min":68.593999999999994,"5min":174.517,"15min":445.44799999999998},"last":23.141999999999999},{"interface":"front","average":{"1min":31.324999999999999,"5min":28.215,"15min":29.988},"min":{"1min":13.523,"5min":10.054,"15min":8.0060000000000002},"max":{"1min":100.619,"5min":205.42500000000001,"15min":400.87599999999998},"last":22.446999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542418,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78916,"kb_used_data":7476,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292924,"statfs":{"total":96636764160,"available":96555954176,"internally_reserved":0,"allocated":7655424,"data_stored":4210307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":9.3670000000000009,"5min":16.529,"15min":18.324000000000002},"min":{"1min":4.7869999999999999,"5min":4.3339999999999996,"15min":4.1840000000000002},"max":{"1min":26.079000000000001,"5min":135.98099999999999,"15min":254.899},"last":12.679},{"interface":"front","average":{"1min":11.316000000000001,"5min":17.039999999999999,"15min":19.542000000000002},"min":{"1min":6.4569999999999999,"5min":5.4219999999999997,"15min":5.4219999999999997},"max":{"1min":28.175999999999998,"5min":97.100999999999999,"15min":270.51600000000002},"last":16.597000000000001}]},{"osd":1,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":19.887,"5min":20.942,"15min":20.899999999999999},"min":{"1min":7.5099999999999998,"5min":7.2350000000000003,"15min":5.1120000000000001},"max":{"1min":138.97300000000001,"5min":149.53899999999999,"15min":177.15799999999999},"last":7.3620000000000001},{"interface":"front","average":{"1min":12.612,"5min":20.626000000000001,"15min":23.800999999999998},"min":{"1min":7.5529999999999999,"5min":6.2199999999999998,"15min":5.883},"max":{"1min":28.498000000000001,"5min":221.483,"15min":309.04199999999997},"last":13.871}]},{"osd":2,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":14.561999999999999,"5min":19.271000000000001,"15min":20.66},"min":{"1min":7.1379999999999999,"5min":6.8929999999999998,"15min":5.859},"max":{"1min":31.204000000000001,"5min":122.018,"15min":231.15600000000001},"last":14.536},{"interface":"front","average":{"1min":16.201000000000001,"5min":24.071999999999999,"15min":23.096},"min":{"1min":11.792,"5min":11.173999999999999,"15min":6.9889999999999999},"max":{"1min":26.43,"5min":149.17500000000001,"15min":197.58699999999999},"last":15.353}]},{"osd":3,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":9.8900000000000006,"5min":253.59399999999999,"15min":99.061000000000007},"min":{"1min":6.6689999999999996,"5min":5.5039999999999996,"15min":5.3849999999999998},"max":{"1min":14.733000000000001,"5min":7958.2420000000002,"15min":7958.2420000000002},"last":5.9400000000000004},{"interface":"front","average":{"1min":13.827,"5min":254.506,"15min":99.292000000000002},"min":{"1min":8.7859999999999996,"5min":6.7009999999999996,"15min":5.9859999999999998},"max":{"1min":23.07,"5min":7971.3900000000003,"15min":7971.3900000000003},"last":8.3620000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":15.43,"5min":25.451000000000001,"15min":25.949000000000002},"min":{"1min":10.375999999999999,"5min":7.5300000000000002,"15min":7.5300000000000002},"max":{"1min":22.405000000000001,"5min":236.994,"15min":262.15699999999998},"last":17.315999999999999},{"interface":"front","average":{"1min":16.157,"5min":23.646999999999998,"15min":23.992000000000001},"min":{"1min":9.6839999999999993,"5min":8.6479999999999997,"15min":6.7859999999999996},"max":{"1min":23.186,"5min":228.88200000000001,"15min":228.88200000000001},"last":10.827999999999999}]},{"osd":5,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":11.866,"5min":20.151,"15min":24.649999999999999},"min":{"1min":6.5540000000000003,"5min":6.2359999999999998,"15min":5.7809999999999997},"max":{"1min":18.652000000000001,"5min":191.154,"15min":289.22699999999998},"last":8.9949999999999992},{"interface":"front","average":{"1min":17.635000000000002,"5min":25.885999999999999,"15min":27.363},"min":{"1min":14.632999999999999,"5min":10.679,"15min":7.835},"max":{"1min":25.047000000000001,"5min":230.613,"15min":230.613},"last":17.914999999999999}]},{"osd":7,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":27.414000000000001,"5min":24.706,"15min":26.167999999999999},"min":{"1min":9.2230000000000008,"5min":8.0370000000000008,"15min":7.2729999999999997},"max":{"1min":164.86600000000001,"5min":192.72399999999999,"15min":290.22300000000001},"last":10.035},{"interface":"front","average":{"1min":27.791,"5min":26.841000000000001,"15min":28.561},"min":{"1min":12.119999999999999,"5min":9.5440000000000005,"15min":7.4489999999999998},"max":{"1min":130.845,"5min":270.01900000000001,"15min":270.01900000000001},"last":15.973000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542418,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60948,"kb_used_data":7552,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310892,"statfs":{"total":96636764160,"available":96574353408,"internally_reserved":0,"allocated":7733248,"data_stored":4297155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":8.5139999999999993,"5min":21.638000000000002,"15min":19.297000000000001},"min":{"1min":4.5659999999999998,"5min":3.7290000000000001,"15min":3.7290000000000001},"max":{"1min":31.369,"5min":278.70299999999997,"15min":317.529},"last":8.1470000000000002},{"interface":"front","average":{"1min":8.5500000000000007,"5min":19.698,"15min":19.140999999999998},"min":{"1min":4.9850000000000003,"5min":4.9850000000000003,"15min":4.8040000000000003},"max":{"1min":32.368000000000002,"5min":174.08099999999999,"15min":200.29300000000001},"last":9.1690000000000005}]},{"osd":1,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":14.672000000000001,"5min":18,"15min":23.431999999999999},"min":{"1min":5.6029999999999998,"5min":5.6029999999999998,"15min":5.6029999999999998},"max":{"1min":25.289999999999999,"5min":136.81200000000001,"15min":307.86799999999999},"last":20.815999999999999},{"interface":"front","average":{"1min":12.586,"5min":18.454999999999998,"15min":23.405000000000001},"min":{"1min":7.9080000000000004,"5min":6.1669999999999998,"15min":6.0819999999999999},"max":{"1min":30.154,"5min":139.191,"15min":307.05500000000001},"last":6.8570000000000002}]},{"osd":2,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":14.393000000000001,"5min":19.882999999999999,"15min":24.52},"min":{"1min":7.2400000000000002,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":25.452999999999999,"5min":279.37099999999998,"15min":283.87599999999998},"last":112.958},{"interface":"front","average":{"1min":24.056000000000001,"5min":21.108000000000001,"15min":22.945},"min":{"1min":9.3710000000000004,"5min":6.9390000000000001,"15min":5.8259999999999996},"max":{"1min":198.02500000000001,"5min":198.02500000000001,"15min":321.54000000000002},"last":92.144999999999996}]},{"osd":3,"last update":"Mon Oct 30 17:54:12 2023","interfaces":[{"interface":"back","average":{"1min":27.785,"5min":165.78,"15min":72.799000000000007},"min":{"1min":9.952,"5min":5.9219999999999997,"15min":5.9219999999999997},"max":{"1min":247.03399999999999,"5min":8219.1820000000007,"15min":8219.1820000000007},"last":22.355},{"interface":"front","average":{"1min":14.07,"5min":161.76900000000001,"15min":71.828000000000003},"min":{"1min":8.7919999999999998,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":34.832999999999998,"5min":8140.4719999999998,"15min":8140.4719999999998},"last":25.198}]},{"osd":5,"last update":"Mon Oct 30 17:54:53 2023","interfaces":[{"interface":"back","average":{"1min":18.893000000000001,"5min":22.899000000000001,"15min":23.814},"min":{"1min":9.1790000000000003,"5min":6.4169999999999998,"15min":6.4169999999999998},"max":{"1min":65.632000000000005,"5min":292.22800000000001,"15min":292.22800000000001},"last":11.455},{"interface":"front","average":{"1min":18.199999999999999,"5min":22.989999999999998,"15min":22.789999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":59.558,"5min":179.67500000000001,"15min":180.28800000000001},"last":12.792999999999999}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":28.52,"5min":26.338999999999999,"15min":25.452000000000002},"min":{"1min":12.355,"5min":9.7889999999999997,"15min":6.5810000000000004},"max":{"1min":191.995,"5min":191.995,"15min":293.16300000000001},"last":12.082000000000001},{"interface":"front","average":{"1min":17.751999999999999,"5min":26.558,"15min":27.359999999999999},"min":{"1min":8.516,"5min":8.2420000000000009,"15min":7.6779999999999999},"max":{"1min":43.220999999999997,"5min":207.30099999999999,"15min":281.62799999999999},"last":23.100000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":18.670000000000002,"5min":26.806999999999999,"15min":28.308},"min":{"1min":12.002000000000001,"5min":8.0289999999999999,"15min":7.4969999999999999},"max":{"1min":43.93,"5min":396.928,"15min":396.928},"last":9.9209999999999994},{"interface":"front","average":{"1min":18.928000000000001,"5min":32.037999999999997,"15min":29.954999999999998},"min":{"1min":10.805999999999999,"5min":8.9809999999999999,"15min":7.0860000000000003},"max":{"1min":46.781999999999996,"5min":366.32499999999999,"15min":366.32499999999999},"last":21.715}]}]},{"osd":5,"up_from":13,"seq":55834575122,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79328,"kb_used_data":7888,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292512,"statfs":{"total":96636764160,"available":96555532288,"internally_reserved":0,"allocated":8077312,"data_stored":4637651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.609999999999999,"5min":14.541,"15min":16.334},"min":{"1min":4.7089999999999996,"5min":3.9910000000000001,"15min":3.9569999999999999},"max":{"1min":83.933999999999997,"5min":109.919,"15min":155.58799999999999},"last":6.1840000000000002},{"interface":"front","average":{"1min":17.244,"5min":16.513999999999999,"15min":18.204999999999998},"min":{"1min":7.4370000000000003,"5min":4.8789999999999996,"15min":4.6900000000000004},"max":{"1min":82.497,"5min":159.01900000000001,"15min":180.94499999999999},"last":15.596}]},{"osd":1,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":10.683999999999999,"5min":18.123999999999999,"15min":20.596},"min":{"1min":5.0970000000000004,"5min":5.0970000000000004,"15min":4.8810000000000002},"max":{"1min":30.02,"5min":228.15199999999999,"15min":233.22800000000001},"last":16.308},{"interface":"front","average":{"1min":14.571,"5min":15.516999999999999,"15min":18.213000000000001},"min":{"1min":8.5969999999999995,"5min":5.8259999999999996,"15min":5.1120000000000001},"max":{"1min":28.818999999999999,"5min":128.197,"15min":278.41300000000001},"last":18.452000000000002}]},{"osd":2,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":16.053999999999998,"5min":16.785,"15min":19.841000000000001},"min":{"1min":7.101,"5min":7.101,"15min":5.8949999999999996},"max":{"1min":30.600999999999999,"5min":135.75999999999999,"15min":234.09299999999999},"last":17.797000000000001},{"interface":"front","average":{"1min":13.555999999999999,"5min":15.548,"15min":20.547000000000001},"min":{"1min":6.657,"5min":6.5510000000000002,"15min":6.5060000000000002},"max":{"1min":27.719999999999999,"5min":87.316999999999993,"15min":277.52300000000002},"last":19.933}]},{"osd":3,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":13.231999999999999,"5min":151.136,"15min":65.766999999999996},"min":{"1min":6.2080000000000002,"5min":6.2080000000000002,"15min":6.2080000000000002},"max":{"1min":48.613999999999997,"5min":6785.9989999999998,"15min":6785.9989999999998},"last":16.978000000000002},{"interface":"front","average":{"1min":16.672999999999998,"5min":151.255,"15min":65.844999999999999},"min":{"1min":6.6870000000000003,"5min":6.6870000000000003,"15min":6.2839999999999998},"max":{"1min":54.656999999999996,"5min":6786.7420000000002,"15min":6786.7420000000002},"last":31.672000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.928999999999998,"15min":22.541},"min":{"1min":7.2590000000000003,"5min":6.9790000000000001,"15min":6.6609999999999996},"max":{"1min":24.433,"5min":143.75800000000001,"15min":286.96699999999998},"last":19.233000000000001},{"interface":"front","average":{"1min":18.093,"5min":23.826000000000001,"15min":26.565000000000001},"min":{"1min":12.166,"5min":12.166,"15min":8.609},"max":{"1min":26.777000000000001,"5min":144.934,"15min":276.17200000000003},"last":21.702000000000002}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.897,"5min":17.413,"15min":21.896000000000001},"min":{"1min":7.2969999999999997,"5min":6.5190000000000001,"15min":6.5190000000000001},"max":{"1min":77.432000000000002,"5min":77.432000000000002,"15min":280.36000000000001},"last":30.943999999999999},{"interface":"front","average":{"1min":16.088000000000001,"5min":21.170999999999999,"15min":26.117000000000001},"min":{"1min":9.4410000000000007,"5min":7.4020000000000001,"15min":7.4020000000000001},"max":{"1min":25.859999999999999,"5min":109.157,"15min":346.47000000000003},"last":32.313000000000002}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":17.742999999999999,"5min":21.178000000000001,"15min":28.771999999999998},"min":{"1min":8.8689999999999998,"5min":8.8689999999999998,"15min":7.6520000000000001},"max":{"1min":54.5,"5min":110.94799999999999,"15min":332.99099999999999},"last":170.19399999999999},{"interface":"front","average":{"1min":20.643000000000001,"5min":22.628,"15min":29.914999999999999},"min":{"1min":11.954000000000001,"5min":10.728,"15min":9.6750000000000007},"max":{"1min":72.408000000000001,"5min":145.00899999999999,"15min":332.64800000000002},"last":38.954000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T17:55:02.623 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T17:55:02.724 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.722+0000 7f4b0cce7640 10 monclient: get_monmap_and_config 2023-10-30T17:55:02.724 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.722+0000 7f4b0cce7640 10 monclient: build_initial_monmap 2023-10-30T17:55:02.724 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.722+0000 7f4b0cce7640 10 monclient: monmap: 2023-10-30T17:55:02.724 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:02.725 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:02.725 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:02.723705+0000 2023-10-30T17:55:02.725 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:02.723705+0000 2023-10-30T17:55:02.725 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:02.726 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.726 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:02.726 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:02.727 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:02.727 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:02.727 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:02.728 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.723+0000 7f4b0cce7640 1 Processor -- start 2023-10-30T17:55:02.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.723+0000 7f4b0cce7640 1 -- start start 2023-10-30T17:55:02.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.723+0000 7f4b0cce7640 10 monclient: init 2023-10-30T17:55:02.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:02.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-10-30T17:55:02.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4b080b7c90 0x7f4b080a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 10 monclient(hunting): picked mon.noname-a con 0x7f4b080b7c90 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:02.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b0813eb60 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:02.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b08144f50 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:02.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b080aa740 con 0x7f4b08144f50 2023-10-30T17:55:02.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b080aa9b0 con 0x7f4b0813eb60 2023-10-30T17:55:02.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:02.737 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b080a9500 con 0x7f4b080b7c90 2023-10-30T17:55:02.737 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:02.737 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0cce7640 10 monclient(hunting): authenticate will time out at 2685.047227s 2023-10-30T17:55:02.737 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b0659c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4b080b7c90 0x7f4b080a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.738 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b06d9d640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f4b08144f50 0x7f4b08145400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59434/0 (socket says 172.21.15.170:59434) 2023-10-30T17:55:02.738 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.724+0000 7f4b06d9d640 1 -- 172.21.15.170:0/3193936372 learned_addr learned my addr 172.21.15.170:0/3193936372 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:02.738 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.725+0000 7f4b0659c640 10 monclient(hunting): get_auth_request con 0x7f4b080b7c90 auth_method 0 2023-10-30T17:55:02.738 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.725+0000 7f4b0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:02.738 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.725+0000 7f4b0659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.725+0000 7f4b0659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.726+0000 7f4b0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:02.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.726+0000 7f4b0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:02.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.726+0000 7f4b0659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:02.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.727+0000 7f4b0559a640 1 -- 172.21.15.170:0/3193936372 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274944407 0 0) 0x7f4b080aa9b0 con 0x7f4b0813eb60 2023-10-30T17:55:02.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.727+0000 7f4b0559a640 10 monclient(hunting): _negotiate 2023-10-30T17:55:02.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.727+0000 7f4b0559a640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.727+0000 7f4b0559a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.727+0000 7f4b0559a640 10 monclient(hunting): my global_id is 21436 2023-10-30T17:55:02.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.727+0000 7f4b0559a640 1 -- 172.21.15.170:0/3193936372 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af0003520 con 0x7f4b0813eb60 2023-10-30T17:55:02.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.728+0000 7f4b0559a640 1 -- 172.21.15.170:0/3193936372 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384410791 0 0) 0x7f4b080aa740 con 0x7f4b08144f50 2023-10-30T17:55:02.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.728+0000 7f4b0559a640 10 monclient(hunting): _negotiate 2023-10-30T17:55:02.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.728+0000 7f4b0559a640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.728+0000 7f4b0559a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.742 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.728+0000 7f4b0559a640 10 monclient(hunting): my global_id is 15996 2023-10-30T17:55:02.742 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.728+0000 7f4b0559a640 1 -- 172.21.15.170:0/3193936372 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b080aa9b0 con 0x7f4b08144f50 2023-10-30T17:55:02.742 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.729+0000 7f4b0659c640 10 monclient(hunting): handle_auth_done global_id 15993 payload 274 2023-10-30T17:55:02.742 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.729+0000 7f4b0659c640 1 -- 172.21.15.170:0/3193936372 >> v1:172.21.15.177:6789/0 conn(0x7f4b0813eb60 legacy=0x7f4b0813efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.742 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.729+0000 7f4b0659c640 1 -- 172.21.15.170:0/3193936372 >> v1:172.21.15.170:6789/0 conn(0x7f4b08144f50 legacy=0x7f4b08145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.743 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.729+0000 7f4b0659c640 10 monclient: _finish_hunting 0 2023-10-30T17:55:02.744 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.729+0000 7f4b0659c640 1 monclient: found mon.noname-a 2023-10-30T17:55:02.744 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.729+0000 7f4b0659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:02.745 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.729+0000 7f4b0659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:02.745 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.729+0000 7f4b0659c640 1 -- 172.21.15.170:0/3193936372 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b080a9900 con 0x7f4b080b7c90 2023-10-30T17:55:02.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.735+0000 7f4b0659c640 1 --2- 172.21.15.170:0/3193936372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b080b7c90 0x7f4b080a8b80 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f4b000097a0 tx=0x7f4b0002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34bf3baf1ad286fd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:02.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 1 -- 172.21.15.170:0/3193936372 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00030610 con 0x7f4b080b7c90 2023-10-30T17:55:02.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:02.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:02.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 10 monclient: dump: 2023-10-30T17:55:02.750 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:02.750 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:02.751 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.751 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.751 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:02.751 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.751 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:02.752 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:02.752 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:02.752 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 10 monclient: _finish_auth 0 2023-10-30T17:55:02.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 10 monclient: _check_auth_tickets 2023-10-30T17:55:02.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:02.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0cce7640 5 monclient: authenticate success, global_id 15993 2023-10-30T17:55:02.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0cce7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:02.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 1 -- 172.21.15.170:0/3193936372 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b00030770 con 0x7f4b080b7c90 2023-10-30T17:55:02.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:02.753 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 1 -- 172.21.15.170:0/3193936372 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00030ae0 con 0x7f4b080b7c90 2023-10-30T17:55:02.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:02.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:02.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0559a640 10 monclient: dump: 2023-10-30T17:55:02.754 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:02.755 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:02.755 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.755 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.755 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:02.755 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.756 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:02.756 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:02.756 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:02.756 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.756 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0cce7640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:02.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.738+0000 7f4b0cce7640 4 set_mon_vals no callback set 2023-10-30T17:55:02.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.739+0000 7f4b0cce7640 10 monclient: shutdown 2023-10-30T17:55:02.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.739+0000 7f4b0cce7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:02.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.739+0000 7f4b0cce7640 1 -- 172.21.15.170:0/3193936372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b080b7c90 msgr2=0x7f4b080a8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.739+0000 7f4b0cce7640 1 --2- 172.21.15.170:0/3193936372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b080b7c90 0x7f4b080a8b80 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f4b000097a0 tx=0x7f4b0002b960 comp rx=0 tx=0).stop 2023-10-30T17:55:02.757 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.739+0000 7f4b0cce7640 1 -- 172.21.15.170:0/3193936372 shutdown_connections 2023-10-30T17:55:02.758 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.739+0000 7f4b0cce7640 1 --2- 172.21.15.170:0/3193936372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b080b7c90 0x7f4b080a8b80 unknown :-1 s=CLOSED pgs=3365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.758 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.739+0000 7f4b0cce7640 1 -- 172.21.15.170:0/3193936372 >> 172.21.15.170:0/3193936372 conn(0x7f4b08071da0 msgr2=0x7f4b080b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:02.758 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.739+0000 7f4b0cce7640 1 -- 172.21.15.170:0/3193936372 shutdown_connections 2023-10-30T17:55:02.758 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.739+0000 7f4b0cce7640 1 -- 172.21.15.170:0/3193936372 wait complete. 2023-10-30T17:55:02.758 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.740+0000 7f4b0cce7640 10 monclient: build_initial_monmap 2023-10-30T17:55:02.759 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.740+0000 7f4b0cce7640 10 monclient: monmap: 2023-10-30T17:55:02.759 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:02.759 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:02.759 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:02.759 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:02.760 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:02.760 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.760 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:02.760 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:02.760 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:02.761 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.740+0000 7f4b0cce7640 1 librados: starting msgr at 2023-10-30T17:55:02.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.740+0000 7f4b0cce7640 1 librados: starting objecter 2023-10-30T17:55:02.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.740+0000 7f4b0cce7640 1 Processor -- start 2023-10-30T17:55:02.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.740+0000 7f4b0cce7640 1 -- start start 2023-10-30T17:55:02.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.740+0000 7f4b0cce7640 1 librados: setting wanted keys 2023-10-30T17:55:02.762 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.740+0000 7f4b0cce7640 1 librados: calling monclient init 2023-10-30T17:55:02.762 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.740+0000 7f4b0cce7640 10 monclient: init 2023-10-30T17:55:02.762 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:02.762 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:55:02.762 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b080b7c90 0x7f4b08142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 10 monclient(hunting): picked mon.noname-c con 0x7f4b080b7c90 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:02.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0813eb60 0x7f4b0813ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 10 monclient(hunting): picked mon.noname-a con 0x7f4b0813eb60 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:02.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b08144f50 0x7f4b08133550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 10 monclient(hunting): picked mon.noname-b con 0x7f4b08144f50 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:02.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:02.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0813c020 con 0x7f4b0813eb60 2023-10-30T17:55:02.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:02.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0813a5c0 con 0x7f4b08144f50 2023-10-30T17:55:02.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:02.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b081343d0 con 0x7f4b080b7c90 2023-10-30T17:55:02.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:02.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.741+0000 7f4b0cce7640 10 monclient(hunting): authenticate will time out at 2685.064563s 2023-10-30T17:55:02.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b05d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0813eb60 0x7f4b0813ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b0659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b080b7c90 0x7f4b08142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b06d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b08144f50 0x7f4b08133550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b05d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0813eb60 0x7f4b0813ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40404/0 (socket says 172.21.15.170:40404) 2023-10-30T17:55:02.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b05d9b640 1 -- 172.21.15.170:0/1984886620 learned_addr learned my addr 172.21.15.170:0/1984886620 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:02.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b05d9b640 10 monclient(hunting): get_auth_request con 0x7f4b0813eb60 auth_method 0 2023-10-30T17:55:02.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:02.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b05d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b05d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b06d9d640 10 monclient(hunting): get_auth_request con 0x7f4b08144f50 auth_method 0 2023-10-30T17:55:02.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:02.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b06d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b06d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b0659c640 10 monclient(hunting): get_auth_request con 0x7f4b080b7c90 auth_method 0 2023-10-30T17:55:02.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:02.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b0659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:02.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b0659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:02.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b05d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:02.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b05d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:02.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b05d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:02.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:02.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:02.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.742+0000 7f4b06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:02.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.743+0000 7f4b0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:02.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.743+0000 7f4b0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:02.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.743+0000 7f4b0659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:02.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.745+0000 7f4b06d9d640 10 monclient(hunting): handle_auth_done global_id 21439 payload 824 2023-10-30T17:55:02.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.745+0000 7f4b06d9d640 1 -- 172.21.15.170:0/1984886620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b080b7c90 msgr2=0x7f4b08142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.745+0000 7f4b06d9d640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b080b7c90 0x7f4b08142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.745+0000 7f4b06d9d640 1 -- 172.21.15.170:0/1984886620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0813eb60 msgr2=0x7f4b0813ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:02.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.745+0000 7f4b06d9d640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0813eb60 0x7f4b0813ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:02.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.745+0000 7f4b06d9d640 10 monclient: _finish_hunting 0 2023-10-30T17:55:02.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.745+0000 7f4b06d9d640 1 monclient: found mon.noname-b 2023-10-30T17:55:02.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.745+0000 7f4b06d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:02.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.745+0000 7f4b06d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:02.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.745+0000 7f4b06d9d640 1 -- 172.21.15.170:0/1984886620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b00009400 con 0x7f4b08144f50 2023-10-30T17:55:02.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.747+0000 7f4b06d9d640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b08144f50 0x7f4b08133550 secure :-1 s=READY pgs=5266 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c340 tx=0x7f4af8001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:02.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.747+0000 7f4af6ffd640 10 client.?.objecter ms_handle_connect 0x7f4b08144f50 2023-10-30T17:55:02.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.747+0000 7f4af6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:02.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 1 -- 172.21.15.170:0/1984886620 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8005ce0 con 0x7f4b08144f50 2023-10-30T17:55:02.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:02.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:02.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 10 monclient: dump: 2023-10-30T17:55:02.775 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:02.775 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:02.776 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.776 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.776 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:02.776 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.776 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:02.777 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:02.777 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:02.777 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 10 monclient: _finish_auth 0 2023-10-30T17:55:02.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:55:02.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:02.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 1 -- 172.21.15.170:0/1984886620 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af8005e40 con 0x7f4b08144f50 2023-10-30T17:55:02.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4b0cce7640 5 monclient: authenticate success, global_id 21439 2023-10-30T17:55:02.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4b0cce7640 10 monclient: _renew_subs 2023-10-30T17:55:02.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4b0cce7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:02.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4b0cce7640 1 -- 172.21.15.170:0/1984886620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b081345f0 con 0x7f4b08144f50 2023-10-30T17:55:02.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:02.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 1 -- 172.21.15.170:0/1984886620 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af80072d0 con 0x7f4b08144f50 2023-10-30T17:55:02.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:02.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:02.779 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4af6ffd640 10 monclient: dump: 2023-10-30T17:55:02.780 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:02.780 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:02.780 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.780 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:02.780 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:02.781 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:02.781 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:02.781 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:02.781 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:02.781 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:02.781 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4b04d99640 4 set_mon_vals no callback set 2023-10-30T17:55:02.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4b0cce7640 10 client.21439.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:02.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4b0cce7640 10 monclient: _renew_subs 2023-10-30T17:55:02.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4b0cce7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:02.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4b0cce7640 1 -- 172.21.15.170:0/1984886620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b08133c70 con 0x7f4b08144f50 2023-10-30T17:55:02.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.751+0000 7f4b0cce7640 1 librados: init done 2023-10-30T17:55:02.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.752+0000 7f4b0cce7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:02.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.752+0000 7f4b0cce7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:02.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.752+0000 7f4b0cce7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:02.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.752+0000 7f4b0cce7640 1 -- 172.21.15.170:0/1984886620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b08067cd0 con 0x7f4b08144f50 2023-10-30T17:55:02.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.992+0000 7f4af6ffd640 1 -- 172.21.15.170:0/1984886620 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4af8007430 con 0x7f4b08144f50 2023-10-30T17:55:02.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.992+0000 7f4af6ffd640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4ae806c850 0x7f4ae806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:02.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.993+0000 7f4b0659c640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4ae806c850 0x7f4ae806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:02.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.993+0000 7f4b0659c640 10 monclient: get_auth_request con 0x7f4ae806c850 auth_method 0 2023-10-30T17:55:02.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.993+0000 7f4b0659c640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4ae806c850 0x7f4ae806ed10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4b00036000 tx=0x7f4b00008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:02.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.993+0000 7f4af6ffd640 10 client.21439.objecter ms_handle_connect 0x7f4ae806c850 2023-10-30T17:55:02.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.994+0000 7f4af6ffd640 1 -- 172.21.15.170:0/1984886620 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(913..913 src has 374..913) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f4af8057d30 con 0x7f4b08144f50 2023-10-30T17:55:02.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.994+0000 7f4af6ffd640 10 client.21439.objecter ms_dispatch 0x7f4b08000b90 osd_map(913..913 src has 374..913) v4 2023-10-30T17:55:02.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.994+0000 7f4af6ffd640 3 client.21439.objecter handle_osd_map got epochs [913,913] > 0 2023-10-30T17:55:02.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.994+0000 7f4af6ffd640 3 client.21439.objecter handle_osd_map decoding full epoch 913 2023-10-30T17:55:02.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:02.995+0000 7f4af6ffd640 20 client.21439.objecter dump_active .. 0 homeless 2023-10-30T17:55:03.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.005+0000 7f4af6ffd640 1 -- 172.21.15.170:0/1984886620 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4af8054070 con 0x7f4b08144f50 2023-10-30T17:55:03.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.005+0000 7f4af6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:03.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.005+0000 7f4af6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:03.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.180+0000 7f4b0cce7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:55:03.181 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.180+0000 7f4b0cce7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:55:03.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.180+0000 7f4b0cce7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:03.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.180+0000 7f4b0cce7640 1 -- 172.21.15.170:0/1984886620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b0813c1e0 con 0x7f4b08144f50 2023-10-30T17:55:03.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.216+0000 7f4af6ffd640 1 -- 172.21.15.170:0/1984886620 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v913) v1 ==== 74+0+17377 (secure 0 0 0) 0x7f4af8021090 con 0x7f4b08144f50 2023-10-30T17:55:03.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.216+0000 7f4af6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T17:55:03.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.216+0000 7f4af6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:55:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:55:03.219 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":913,"fsid":"dbedaa2b-1947-40d2-95bd-9b4fc7298a39","created":"2023-10-30T17:31:38.447169+0000","modified":"2023-10-30T17:55:00.211483+0000","last_up_change":"2023-10-30T17:32:27.945525+0000","last_in_change":"2023-10-30T17:31:43.906314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":100,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T17:32:25.610766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T17:32:32.227425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-10-30T17:33:34.045100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"26bebf29-e72f-410f-aff7-561160c246e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6809","nonce":500152820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6811","nonce":500152820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6815","nonce":500152820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":500152820},{"type":"v1","addr":"172.21.15.170:6813","nonce":500152820}]},"public_addr":"172.21.15.170:6809/500152820","cluster_addr":"172.21.15.170:6811/500152820","heartbeat_back_addr":"172.21.15.170:6815/500152820","heartbeat_front_addr":"172.21.15.170:6813/500152820","state":["exists","up"]},{"osd":1,"uuid":"bc8beb5c-4ab4-4e03-a2ea-fa0a11eb55be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6825","nonce":527201623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6827","nonce":527201623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6831","nonce":527201623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":527201623},{"type":"v1","addr":"172.21.15.170:6829","nonce":527201623}]},"public_addr":"172.21.15.170:6825/527201623","cluster_addr":"172.21.15.170:6827/527201623","heartbeat_back_addr":"172.21.15.170:6831/527201623","heartbeat_front_addr":"172.21.15.170:6829/527201623","state":["exists","up"]},{"osd":2,"uuid":"8f70bfd6-88a2-413b-9663-f9bfed6ebe74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6817","nonce":425342180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6819","nonce":425342180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6823","nonce":425342180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":425342180},{"type":"v1","addr":"172.21.15.170:6821","nonce":425342180}]},"public_addr":"172.21.15.170:6817/425342180","cluster_addr":"172.21.15.170:6819/425342180","heartbeat_back_addr":"172.21.15.170:6823/425342180","heartbeat_front_addr":"172.21.15.170:6821/425342180","state":["exists","up"]},{"osd":3,"uuid":"46c2dd9a-c073-42ab-a07b-918f83b8f7f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6801","nonce":3657546070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6803","nonce":3657546070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6807","nonce":3657546070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3657546070},{"type":"v1","addr":"172.21.15.170:6805","nonce":3657546070}]},"public_addr":"172.21.15.170:6801/3657546070","cluster_addr":"172.21.15.170:6803/3657546070","heartbeat_back_addr":"172.21.15.170:6807/3657546070","heartbeat_front_addr":"172.21.15.170:6805/3657546070","state":["exists","up"]},{"osd":4,"uuid":"94b9a549-2c12-42bb-b3f1-2e3304c574e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":901,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6803","nonce":2107882913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6805","nonce":2107882913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6809","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6811","nonce":2107882913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2107882913},{"type":"v1","addr":"172.21.15.177:6807","nonce":2107882913}]},"public_addr":"172.21.15.177:6803/2107882913","cluster_addr":"172.21.15.177:6805/2107882913","heartbeat_back_addr":"172.21.15.177:6811/2107882913","heartbeat_front_addr":"172.21.15.177:6807/2107882913","state":["exists","up"]},{"osd":5,"uuid":"2314fc35-d5d3-4609-853a-c0cbf5212ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6810","nonce":2243649533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6813","nonce":2243649533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6817","nonce":2243649533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2243649533},{"type":"v1","addr":"172.21.15.177:6815","nonce":2243649533}]},"public_addr":"172.21.15.177:6810/2243649533","cluster_addr":"172.21.15.177:6813/2243649533","heartbeat_back_addr":"172.21.15.177:6817/2243649533","heartbeat_front_addr":"172.21.15.177:6815/2243649533","state":["exists","up"]},{"osd":6,"uuid":"3e78cc0a-2199-401d-b956-382ab3af2ed1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6827","nonce":1041798931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6829","nonce":1041798931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6833","nonce":1041798931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1041798931},{"type":"v1","addr":"172.21.15.177:6831","nonce":1041798931}]},"public_addr":"172.21.15.177:6827/1041798931","cluster_addr":"172.21.15.177:6829/1041798931","heartbeat_back_addr":"172.21.15.177:6833/1041798931","heartbeat_front_addr":"172.21.15.177:6831/1041798931","state":["exists","up"]},{"osd":7,"uuid":"7388842d-4347-4fd7-b258-09c16109727d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6819","nonce":3145923709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6821","nonce":3145923709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6825","nonce":3145923709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3145923709},{"type":"v1","addr":"172.21.15.177:6823","nonce":3145923709}]},"public_addr":"172.21.15.177:6819/3145923709","cluster_addr":"172.21.15.177:6821/3145923709","heartbeat_back_addr":"172.21.15.177:6825/3145923709","heartbeat_front_addr":"172.21.15.177:6823/3145923709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:24.231313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:24.066267+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:23.315345+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:23.639773+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.166624+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.275780+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:21.364855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T17:32:22.235136+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":9,"length":2}]},{"pool":16,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":55,"snaps":[{"begin":11,"length":2}]},{"pool":82,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":93,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":110,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":116,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":176,"snaps":[{"begin":12,"length":1}]},{"pool":186,"snaps":[{"begin":14,"length":1}]},{"pool":191,"snaps":[{"begin":17,"length":1}]},{"pool":193,"snaps":[{"begin":19,"length":2}]},{"pool":197,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T17:55:03.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.218+0000 7f4b0cce7640 10 librados: watch_flush enter 2023-10-30T17:55:03.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.218+0000 7f4b0cce7640 10 librados: watch_flush exit 2023-10-30T17:55:03.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.218+0000 7f4b0cce7640 20 client.21439.objecter shutdown clearing up homeless session... 2023-10-30T17:55:03.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.218+0000 7f4b0cce7640 10 client.21439.objecter successfully canceled tick 2023-10-30T17:55:03.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 -- 172.21.15.170:0/1984886620 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4ae806c850 msgr2=0x7f4ae806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:03.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4ae806c850 0x7f4ae806ed10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4b00036000 tx=0x7f4b00008030 comp rx=0 tx=0).stop 2023-10-30T17:55:03.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 10 monclient: shutdown 2023-10-30T17:55:03.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:03.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 -- 172.21.15.170:0/1984886620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b08144f50 msgr2=0x7f4b08133550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:03.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b08144f50 0x7f4b08133550 secure :-1 s=READY pgs=5266 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c340 tx=0x7f4af8001e70 comp rx=0 tx=0).stop 2023-10-30T17:55:03.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 -- 172.21.15.170:0/1984886620 shutdown_connections 2023-10-30T17:55:03.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f4ae806c850 0x7f4ae806ed10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:03.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b08144f50 0x7f4b08133550 unknown :-1 s=CLOSED pgs=5266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:03.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0813eb60 0x7f4b0813ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:03.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 --2- 172.21.15.170:0/1984886620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b080b7c90 0x7f4b08142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:03.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 -- 172.21.15.170:0/1984886620 >> 172.21.15.170:0/1984886620 conn(0x7f4b08071da0 msgr2=0x7f4b080b23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:03.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 -- 172.21.15.170:0/1984886620 shutdown_connections 2023-10-30T17:55:03.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 -- 172.21.15.170:0/1984886620 wait complete. 2023-10-30T17:55:03.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:03.219+0000 7f4b0cce7640 1 librados: shutdown 2023-10-30T17:55:04.232 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T17:55:04.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T17:55:04.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.350+0000 7f424fa44640 10 monclient: get_monmap_and_config 2023-10-30T17:55:04.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.350+0000 7f424fa44640 10 monclient: build_initial_monmap 2023-10-30T17:55:04.352 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.350+0000 7f424fa44640 10 monclient: monmap: 2023-10-30T17:55:04.352 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:04.353 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:04.353 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:04.351436+0000 2023-10-30T17:55:04.353 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:04.351436+0000 2023-10-30T17:55:04.353 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:04.354 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.354 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:04.354 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:04.354 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:04.354 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:04.355 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:04.355 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 1 Processor -- start 2023-10-30T17:55:04.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 1 -- start start 2023-10-30T17:55:04.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 10 monclient: init 2023-10-30T17:55:04.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:04.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-10-30T17:55:04.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f42480b4460 0x7f42480ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 10 monclient(hunting): picked mon.noname-a con 0x7f42480b4460 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:04.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f424813eb10 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:04.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f42480b08c0 0x7f42480b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.359 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 10 monclient(hunting): picked mon.noname-c con 0x7f42480b08c0 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:04.359 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42480aeda0 con 0x7f424813eb10 2023-10-30T17:55:04.359 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:04.359 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42480b4840 con 0x7f42480b4460 2023-10-30T17:55:04.360 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:04.360 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42480b04a0 con 0x7f42480b08c0 2023-10-30T17:55:04.360 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:04.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.351+0000 7f424fa44640 10 monclient(hunting): authenticate will time out at 2686.674986s 2023-10-30T17:55:04.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.352+0000 7f424cfdf640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f424813eb10 0x7f424813ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:45902/0 (socket says 172.21.15.170:45902) 2023-10-30T17:55:04.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.352+0000 7f424cfdf640 1 -- 172.21.15.170:0/1675315367 learned_addr learned my addr 172.21.15.170:0/1675315367 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:04.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.352+0000 7f424dfe1640 1 --2- 172.21.15.170:0/1675315367 >> v2:172.21.15.177:3300/0 conn(0x7f42480b08c0 0x7f42480b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:04.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.353+0000 7f424dfe1640 10 monclient(hunting): get_auth_request con 0x7f42480b08c0 auth_method 0 2023-10-30T17:55:04.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.353+0000 7f424dfe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:04.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.353+0000 7f424dfe1640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:04.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.353+0000 7f424dfe1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:04.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.353+0000 7f424dfe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:04.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.353+0000 7f424dfe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:04.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.353+0000 7f424dfe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:04.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.356+0000 7f424dfe1640 10 monclient(hunting): handle_auth_done global_id 21442 payload 274 2023-10-30T17:55:04.364 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.356+0000 7f424dfe1640 1 -- 172.21.15.170:0/1675315367 >> v2:172.21.15.170:3300/0 conn(0x7f42480b4460 msgr2=0x7f42480ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:04.364 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.356+0000 7f424dfe1640 1 --2- 172.21.15.170:0/1675315367 >> v2:172.21.15.170:3300/0 conn(0x7f42480b4460 0x7f42480ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:04.364 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.356+0000 7f424dfe1640 1 -- 172.21.15.170:0/1675315367 >> v1:172.21.15.177:6789/0 conn(0x7f424813eb10 legacy=0x7f424813ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.364 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.356+0000 7f424dfe1640 10 monclient: _finish_hunting 0 2023-10-30T17:55:04.365 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.356+0000 7f424dfe1640 1 monclient: found mon.noname-c 2023-10-30T17:55:04.365 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.356+0000 7f424dfe1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:04.365 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.356+0000 7f424dfe1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:55:04.365 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.356+0000 7f424dfe1640 1 -- 172.21.15.170:0/1675315367 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42480af010 con 0x7f42480b08c0 2023-10-30T17:55:04.365 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.359+0000 7f424dfe1640 1 --2- 172.21.15.170:0/1675315367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42480b08c0 0x7f42480b0d30 secure :-1 s=READY pgs=5268 cs=0 l=1 rev1=1 crypto rx=0x7f4248142a00 tx=0x7f424002be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e6b3789e86b75cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:04.368 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 1 -- 172.21.15.170:0/1675315367 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424002cbf0 con 0x7f42480b08c0 2023-10-30T17:55:04.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:04.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:04.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 10 monclient: dump: 2023-10-30T17:55:04.369 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:04.369 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:04.370 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.370 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.370 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:04.370 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.371 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:04.371 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:04.371 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:04.371 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 10 monclient: _finish_auth 0 2023-10-30T17:55:04.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 10 monclient: _check_auth_tickets 2023-10-30T17:55:04.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:04.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 1 -- 172.21.15.170:0/1675315367 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424002cd50 con 0x7f42480b08c0 2023-10-30T17:55:04.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:04.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 1 -- 172.21.15.170:0/1675315367 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240034660 con 0x7f42480b08c0 2023-10-30T17:55:04.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f424fa44640 5 monclient: authenticate success, global_id 21442 2023-10-30T17:55:04.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f424fa44640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:04.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f424fa44640 4 set_mon_vals no callback set 2023-10-30T17:55:04.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:04.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:04.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f423ffff640 10 monclient: dump: 2023-10-30T17:55:04.374 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:04.374 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:04.374 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:04.375 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:04.376 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:04.376 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f424fa44640 10 monclient: shutdown 2023-10-30T17:55:04.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f424fa44640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:04.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f424fa44640 1 -- 172.21.15.170:0/1675315367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42480b08c0 msgr2=0x7f42480b0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.363+0000 7f424fa44640 1 --2- 172.21.15.170:0/1675315367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42480b08c0 0x7f42480b0d30 secure :-1 s=READY pgs=5268 cs=0 l=1 rev1=1 crypto rx=0x7f4248142a00 tx=0x7f424002be40 comp rx=0 tx=0).stop 2023-10-30T17:55:04.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.364+0000 7f424fa44640 1 -- 172.21.15.170:0/1675315367 shutdown_connections 2023-10-30T17:55:04.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.364+0000 7f424fa44640 1 --2- 172.21.15.170:0/1675315367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42480b08c0 0x7f42480b0d30 unknown :-1 s=CLOSED pgs=5268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:04.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.364+0000 7f424fa44640 1 --2- 172.21.15.170:0/1675315367 >> v2:172.21.15.170:3300/0 conn(0x7f42480b4460 0x7f42480ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.364+0000 7f424fa44640 1 -- 172.21.15.170:0/1675315367 >> 172.21.15.170:0/1675315367 conn(0x7f4248071b90 msgr2=0x7f4248071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.364+0000 7f424fa44640 1 -- 172.21.15.170:0/1675315367 shutdown_connections 2023-10-30T17:55:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.364+0000 7f424fa44640 1 -- 172.21.15.170:0/1675315367 wait complete. 2023-10-30T17:55:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.364+0000 7f424fa44640 10 monclient: build_initial_monmap 2023-10-30T17:55:04.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.364+0000 7f424fa44640 10 monclient: monmap: 2023-10-30T17:55:04.379 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:04.379 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:04.379 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:04.380 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:04.380 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:04.380 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.380 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:04.380 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:04.380 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:04.381 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 librados: starting msgr at 2023-10-30T17:55:04.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 librados: starting objecter 2023-10-30T17:55:04.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 Processor -- start 2023-10-30T17:55:04.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 -- start start 2023-10-30T17:55:04.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 librados: setting wanted keys 2023-10-30T17:55:04.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 librados: calling monclient init 2023-10-30T17:55:04.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient: init 2023-10-30T17:55:04.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:04.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:55:04.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42480b08c0 0x7f42481cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient(hunting): picked mon.noname-a con 0x7f42480b08c0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:04.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42480b4460 0x7f42481d7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient(hunting): picked mon.noname-c con 0x7f42480b4460 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:04.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f424813eb10 0x7f42481d9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient(hunting): picked mon.noname-b con 0x7f424813eb10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:04.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:04.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42481dac30 con 0x7f42480b08c0 2023-10-30T17:55:04.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:04.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42481395d0 con 0x7f424813eb10 2023-10-30T17:55:04.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:04.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42481d6d50 con 0x7f42480b4460 2023-10-30T17:55:04.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:04.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.365+0000 7f424fa44640 10 monclient(hunting): authenticate will time out at 2686.688793s 2023-10-30T17:55:04.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424dfe1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f424813eb10 0x7f42481d9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:04.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424cfdf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42480b4460 0x7f42481d7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:04.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424dfe1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f424813eb10 0x7f42481d9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:49482/0 (socket says 172.21.15.170:49482) 2023-10-30T17:55:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424dfe1640 1 -- 172.21.15.170:0/3876658827 learned_addr learned my addr 172.21.15.170:0/3876658827 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424dfe1640 10 monclient(hunting): get_auth_request con 0x7f424813eb10 auth_method 0 2023-10-30T17:55:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424dfe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424dfe1640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424dfe1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424cfdf640 10 monclient(hunting): get_auth_request con 0x7f42480b4460 auth_method 0 2023-10-30T17:55:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424cfdf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424cfdf640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424cfdf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424dfe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424dfe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.366+0000 7f424dfe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.367+0000 7f424cfdf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.367+0000 7f424cfdf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.367+0000 7f424cfdf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:04.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.369+0000 7f424dfe1640 10 monclient(hunting): handle_auth_done global_id 21445 payload 824 2023-10-30T17:55:04.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.369+0000 7f424dfe1640 1 -- 172.21.15.170:0/3876658827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42480b4460 msgr2=0x7f42481d7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.369+0000 7f424dfe1640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42480b4460 0x7f42481d7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:04.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.369+0000 7f424dfe1640 1 -- 172.21.15.170:0/3876658827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42480b08c0 msgr2=0x7f42481cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.369+0000 7f424dfe1640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42480b08c0 0x7f42481cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:04.391 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.369+0000 7f424dfe1640 10 monclient: _finish_hunting 0 2023-10-30T17:55:04.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.369+0000 7f424dfe1640 1 monclient: found mon.noname-b 2023-10-30T17:55:04.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.369+0000 7f424dfe1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:04.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.369+0000 7f424dfe1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:04.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.369+0000 7f424dfe1640 1 -- 172.21.15.170:0/3876658827 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4240009a70 con 0x7f424813eb10 2023-10-30T17:55:04.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.371+0000 7f424dfe1640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f424813eb10 0x7f42481d9ce0 secure :-1 s=READY pgs=5269 cs=0 l=1 rev1=1 crypto rx=0x7f4240037040 tx=0x7f4240004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:04.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.371+0000 7f423dffb640 10 client.?.objecter ms_handle_connect 0x7f424813eb10 2023-10-30T17:55:04.393 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.372+0000 7f423dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:04.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.426+0000 7f423dffb640 1 -- 172.21.15.170:0/3876658827 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240040650 con 0x7f424813eb10 2023-10-30T17:55:04.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.426+0000 7f423dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:04.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:04.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 10 monclient: dump: 2023-10-30T17:55:04.430 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:04.430 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:04.430 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.430 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.430 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:04.431 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.431 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:04.431 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:04.431 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:04.431 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 10 monclient: _finish_auth 0 2023-10-30T17:55:04.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 10 monclient: _check_auth_tickets 2023-10-30T17:55:04.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:04.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 1 -- 172.21.15.170:0/3876658827 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42400407b0 con 0x7f424813eb10 2023-10-30T17:55:04.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f424fa44640 5 monclient: authenticate success, global_id 21445 2023-10-30T17:55:04.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f424fa44640 10 monclient: _renew_subs 2023-10-30T17:55:04.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f424fa44640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:04.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42481da220 con 0x7f424813eb10 2023-10-30T17:55:04.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:04.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 1 -- 172.21.15.170:0/3876658827 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240040b20 con 0x7f424813eb10 2023-10-30T17:55:04.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:04.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:04.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423dffb640 10 monclient: dump: 2023-10-30T17:55:04.434 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:04.434 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:04.434 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.435 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.435 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:04.435 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.435 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:04.436 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:04.436 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:04.436 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f423f7fe640 4 set_mon_vals no callback set 2023-10-30T17:55:04.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f424fa44640 10 client.21445.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:04.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f424fa44640 10 monclient: _renew_subs 2023-10-30T17:55:04.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f424fa44640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:04.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42481d69c0 con 0x7f424813eb10 2023-10-30T17:55:04.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.427+0000 7f424fa44640 1 librados: init done 2023-10-30T17:55:04.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.428+0000 7f424fa44640 10 client.21445.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:04.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.428+0000 7f424fa44640 15 client.21445.objecter _session_command_op_assign -1 1 2023-10-30T17:55:04.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.428+0000 7f424fa44640 20 client.21445.objecter _get_session osd=-1 returning homeless 2023-10-30T17:55:04.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.428+0000 7f424fa44640 10 client.21445.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:04.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.428+0000 7f424fa44640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:04.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.428+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f42480b3a70 con 0x7f424813eb10 2023-10-30T17:55:04.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.446+0000 7f423dffb640 1 -- 172.21.15.170:0/3876658827 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4240040c80 con 0x7f424813eb10 2023-10-30T17:55:04.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.446+0000 7f423dffb640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f422c06c850 0x7f422c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.447+0000 7f424d7e0640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f422c06c850 0x7f422c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:04.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.447+0000 7f424d7e0640 10 monclient: get_auth_request con 0x7f422c06c850 auth_method 0 2023-10-30T17:55:04.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.447+0000 7f424d7e0640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f422c06c850 0x7f422c06ed10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4244009710 tx=0x7f4244006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:04.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.447+0000 7f423dffb640 10 client.21445.objecter ms_handle_connect 0x7f422c06c850 2023-10-30T17:55:04.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 1 -- 172.21.15.170:0/3876658827 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f424007c000 con 0x7f424813eb10 2023-10-30T17:55:04.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 10 client.21445.objecter ms_dispatch 0x7f4248000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:04.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 3 client.21445.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:04.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 3 client.21445.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:04.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 10 client.21445.objecter checking command 1 2023-10-30T17:55:04.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] conn(0x7f422c074060 0x7f422c076500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 20 client.21445.objecter _get_session s=0x7f422c073990 osd=0 3 2023-10-30T17:55:04.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 20 client.21445.objecter put_session s=0x7f422c073990 osd=0 3 2023-10-30T17:55:04.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 15 client.21445.objecter _session_command_op_remove -1 1 2023-10-30T17:55:04.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 20 client.21445.objecter _get_session s=0x7f422c073990 osd=0 3 2023-10-30T17:55:04.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 20 client.21445.objecter get_session s=0x7f422c073990 osd=0 3 2023-10-30T17:55:04.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 15 client.21445.objecter _session_command_op_assign 0 1 2023-10-30T17:55:04.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 20 client.21445.objecter put_session s=0x7f422c073990 osd=0 4 2023-10-30T17:55:04.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 10 client.21445.objecter _send_command 1 2023-10-30T17:55:04.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 1 -- 172.21.15.170:0/3876658827 --> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f422c076bd0 con 0x7f422c074060 2023-10-30T17:55:04.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f423dffb640 20 client.21445.objecter dump_active .. 0 homeless 2023-10-30T17:55:04.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.449+0000 7f424cfdf640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] conn(0x7f422c074060 0x7f422c076500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:04.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.450+0000 7f424cfdf640 10 monclient: get_auth_request con 0x7f422c074060 auth_method 0 2023-10-30T17:55:04.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.450+0000 7f423dffb640 1 -- 172.21.15.170:0/3876658827 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=914) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4240077c40 con 0x7f424813eb10 2023-10-30T17:55:04.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.450+0000 7f423dffb640 10 monclient: handle_get_version_reply finishing 1 version 914 2023-10-30T17:55:04.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.452+0000 7f423dffb640 10 client.21445.objecter ms_handle_connect 0x7f422c074060 2023-10-30T17:55:04.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.452+0000 7f424cfdf640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] conn(0x7f422c074060 0x7f422c076500 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:04.476 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.475+0000 7f423dffb640 1 -- 172.21.15.170:0/3876658827 <== osd.0 v2:172.21.15.170:6808/500152820 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f422c076bd0 con 0x7f422c074060 2023-10-30T17:55:04.476 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.475+0000 7f423dffb640 10 client.21445.objecter ms_dispatch 0x7f4248000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:55:04.476 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.475+0000 7f423dffb640 10 client.21445.objecter _finish_command 1 = system:0 2023-10-30T17:55:04.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.475+0000 7f423dffb640 20 client.21445.objecter put_session s=0x7f422c073990 osd=0 4 2023-10-30T17:55:04.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.475+0000 7f423dffb640 15 client.21445.objecter _session_command_op_remove 0 1 2023-10-30T17:55:04.495 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 10 client.21445.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T17:55:04.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 15 client.21445.objecter _session_command_op_assign -1 2 2023-10-30T17:55:04.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 20 client.21445.objecter _get_session s=0x7f422c073990 osd=0 3 2023-10-30T17:55:04.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 20 client.21445.objecter put_session s=0x7f422c073990 osd=0 3 2023-10-30T17:55:04.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 20 client.21445.objecter _get_session s=0x7f422c073990 osd=0 3 2023-10-30T17:55:04.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 15 client.21445.objecter _session_command_op_remove -1 2 2023-10-30T17:55:04.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 20 client.21445.objecter get_session s=0x7f422c073990 osd=0 3 2023-10-30T17:55:04.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 15 client.21445.objecter _session_command_op_assign 0 2 2023-10-30T17:55:04.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 20 client.21445.objecter put_session s=0x7f422c073990 osd=0 4 2023-10-30T17:55:04.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 10 client.21445.objecter _send_command 2 2023-10-30T17:55:04.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.494+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 --> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4248067cd0 con 0x7f422c074060 2023-10-30T17:55:04.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f423dffb640 1 -- 172.21.15.170:0/3876658827 <== osd.0 v2:172.21.15.170:6808/500152820 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4248067cd0 con 0x7f422c074060 2023-10-30T17:55:04.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f423dffb640 10 client.21445.objecter ms_dispatch 0x7f4248000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:55:04.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f423dffb640 10 client.21445.objecter _finish_command 2 = system:0 2023-10-30T17:55:04.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f423dffb640 20 client.21445.objecter put_session s=0x7f422c073990 osd=0 4 2023-10-30T17:55:04.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f423dffb640 15 client.21445.objecter _session_command_op_remove 0 2 2023-10-30T17:55:04.511 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-30T17:55:04.512 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T17:55:04.512 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-30T17:55:04.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 10 librados: watch_flush enter 2023-10-30T17:55:04.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 10 librados: watch_flush exit 2023-10-30T17:55:04.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 10 client.21445.objecter close_session for osd.0 2023-10-30T17:55:04.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 >> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] conn(0x7f422c074060 msgr2=0x7f422c076500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.514 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.170:6808/500152820,v1:172.21.15.170:6809/500152820] conn(0x7f422c074060 0x7f422c076500 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:04.515 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 20 client.21445.objecter put_session s=0x7f422c073990 osd=0 1 2023-10-30T17:55:04.515 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 20 client.21445.objecter shutdown clearing up homeless session... 2023-10-30T17:55:04.515 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 10 client.21445.objecter successfully canceled tick 2023-10-30T17:55:04.515 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f422c06c850 msgr2=0x7f422c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.515 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f422c06c850 0x7f422c06ed10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4244009710 tx=0x7f4244006c90 comp rx=0 tx=0).stop 2023-10-30T17:55:04.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 10 monclient: shutdown 2023-10-30T17:55:04.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:04.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f424813eb10 msgr2=0x7f42481d9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 --2- 172.21.15.170:0/3876658827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f424813eb10 0x7f42481d9ce0 secure :-1 s=READY pgs=5269 cs=0 l=1 rev1=1 crypto rx=0x7f4240037040 tx=0x7f4240004a40 comp rx=0 tx=0).stop 2023-10-30T17:55:04.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424dfe1640 1 -- 172.21.15.170:0/3876658827 reap_dead start 2023-10-30T17:55:04.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 shutdown_connections 2023-10-30T17:55:04.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 >> 172.21.15.170:0/3876658827 conn(0x7f4248071b90 msgr2=0x7f42481c8300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:04.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 shutdown_connections 2023-10-30T17:55:04.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 -- 172.21.15.170:0/3876658827 wait complete. 2023-10-30T17:55:04.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.509+0000 7f424fa44640 1 librados: shutdown 2023-10-30T17:55:04.521 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T17:55:04.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.621+0000 7f6b2433e640 10 monclient: get_monmap_and_config 2023-10-30T17:55:04.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.621+0000 7f6b2433e640 10 monclient: build_initial_monmap 2023-10-30T17:55:04.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.621+0000 7f6b2433e640 10 monclient: monmap: 2023-10-30T17:55:04.623 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:04.623 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:04.624 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:04.622110+0000 2023-10-30T17:55:04.624 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:04.622110+0000 2023-10-30T17:55:04.624 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:04.624 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.625 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:04.625 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:04.625 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:04.625 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:04.625 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:04.626 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.621+0000 7f6b2433e640 1 Processor -- start 2023-10-30T17:55:04.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.621+0000 7f6b2433e640 1 -- start start 2023-10-30T17:55:04.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.621+0000 7f6b2433e640 10 monclient: init 2023-10-30T17:55:04.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:04.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-10-30T17:55:04.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b1c0b7c90 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:04.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b1c13eb60 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:04.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6b1c144f50 0x7f6b1c145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 10 monclient(hunting): picked mon.noname-c con 0x7f6b1c144f50 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:04.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1c0aa740 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1c0aa9b0 con 0x7f6b1c13eb60 2023-10-30T17:55:04.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:04.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b1c0a9500 con 0x7f6b1c144f50 2023-10-30T17:55:04.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:04.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.622+0000 7f6b2433e640 10 monclient(hunting): authenticate will time out at 2686.945657s 2023-10-30T17:55:04.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.623+0000 7f6b220da640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f6b1c0b7c90 0x7f6b1c0a8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59438/0 (socket says 172.21.15.170:59438) 2023-10-30T17:55:04.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.623+0000 7f6b220da640 1 -- 172.21.15.170:0/2240577774 learned_addr learned my addr 172.21.15.170:0/2240577774 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:04.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.623+0000 7f6b228db640 1 --2- 172.21.15.170:0/2240577774 >> v2:172.21.15.177:3300/0 conn(0x7f6b1c144f50 0x7f6b1c145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:04.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.623+0000 7f6b228db640 10 monclient(hunting): get_auth_request con 0x7f6b1c144f50 auth_method 0 2023-10-30T17:55:04.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.623+0000 7f6b228db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:04.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.623+0000 7f6b228db640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:04.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.623+0000 7f6b228db640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:04.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.624+0000 7f6b228db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:04.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.624+0000 7f6b228db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:04.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.624+0000 7f6b228db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:04.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.625+0000 7f6b210d8640 1 -- 172.21.15.170:0/2240577774 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151298278 0 0) 0x7f6b1c0aa740 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.625+0000 7f6b210d8640 10 monclient(hunting): _negotiate 2023-10-30T17:55:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.625+0000 7f6b210d8640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.625+0000 7f6b210d8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.625+0000 7f6b210d8640 10 monclient(hunting): my global_id is 16002 2023-10-30T17:55:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.625+0000 7f6b210d8640 1 -- 172.21.15.170:0/2240577774 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b08003520 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.626+0000 7f6b228db640 10 monclient(hunting): handle_auth_done global_id 21448 payload 274 2023-10-30T17:55:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.626+0000 7f6b228db640 1 -- 172.21.15.170:0/2240577774 >> v1:172.21.15.177:6789/0 conn(0x7f6b1c13eb60 legacy=0x7f6b1c13efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.626+0000 7f6b228db640 1 -- 172.21.15.170:0/2240577774 >> v1:172.21.15.170:6789/0 conn(0x7f6b1c0b7c90 legacy=0x7f6b1c0a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.626+0000 7f6b228db640 10 monclient: _finish_hunting 0 2023-10-30T17:55:04.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.626+0000 7f6b228db640 1 monclient: found mon.noname-c 2023-10-30T17:55:04.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.626+0000 7f6b228db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:04.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.626+0000 7f6b228db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:55:04.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.626+0000 7f6b228db640 1 -- 172.21.15.170:0/2240577774 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c0a9900 con 0x7f6b1c144f50 2023-10-30T17:55:04.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.628+0000 7f6b228db640 1 --2- 172.21.15.170:0/2240577774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b1c144f50 0x7f6b1c145400 secure :-1 s=READY pgs=5271 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009d90 tx=0x7f6b1802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2775e4843c0445ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:04.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 1 -- 172.21.15.170:0/2240577774 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1802cc60 con 0x7f6b1c144f50 2023-10-30T17:55:04.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:04.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:04.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 10 monclient: dump: 2023-10-30T17:55:04.643 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:04.644 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:04.644 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.644 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.644 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:04.645 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.645 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:04.645 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:04.646 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:04.646 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 10 monclient: _finish_auth 0 2023-10-30T17:55:04.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 10 monclient: _check_auth_tickets 2023-10-30T17:55:04.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:04.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 1 -- 172.21.15.170:0/2240577774 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1802cdc0 con 0x7f6b1c144f50 2023-10-30T17:55:04.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:04.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 1 -- 172.21.15.170:0/2240577774 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18034660 con 0x7f6b1c144f50 2023-10-30T17:55:04.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b2433e640 5 monclient: authenticate success, global_id 21448 2023-10-30T17:55:04.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b2433e640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:04.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b2433e640 4 set_mon_vals no callback set 2023-10-30T17:55:04.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:04.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:04.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.633+0000 7f6b210d8640 10 monclient: dump: 2023-10-30T17:55:04.648 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:04.649 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:04.649 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.649 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.649 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:04.650 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.650 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:04.650 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:04.651 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:04.651 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.634+0000 7f6b2433e640 10 monclient: shutdown 2023-10-30T17:55:04.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.634+0000 7f6b2433e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:04.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.634+0000 7f6b2433e640 1 -- 172.21.15.170:0/2240577774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b1c144f50 msgr2=0x7f6b1c145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.634+0000 7f6b2433e640 1 --2- 172.21.15.170:0/2240577774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b1c144f50 0x7f6b1c145400 secure :-1 s=READY pgs=5271 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009d90 tx=0x7f6b1802beb0 comp rx=0 tx=0).stop 2023-10-30T17:55:04.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.634+0000 7f6b2433e640 1 -- 172.21.15.170:0/2240577774 shutdown_connections 2023-10-30T17:55:04.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.634+0000 7f6b2433e640 1 --2- 172.21.15.170:0/2240577774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b1c144f50 0x7f6b1c145400 unknown :-1 s=CLOSED pgs=5271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:04.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.634+0000 7f6b2433e640 1 -- 172.21.15.170:0/2240577774 >> 172.21.15.170:0/2240577774 conn(0x7f6b1c071da0 msgr2=0x7f6b1c0b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:04.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.634+0000 7f6b2433e640 1 -- 172.21.15.170:0/2240577774 shutdown_connections 2023-10-30T17:55:04.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.634+0000 7f6b2433e640 1 -- 172.21.15.170:0/2240577774 wait complete. 2023-10-30T17:55:04.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 10 monclient: build_initial_monmap 2023-10-30T17:55:04.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 10 monclient: monmap: 2023-10-30T17:55:04.653 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:04.654 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:04.654 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:04.654 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:04.654 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:04.655 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.655 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:04.655 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:04.656 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:04.656 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 1 librados: starting msgr at 2023-10-30T17:55:04.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 1 librados: starting objecter 2023-10-30T17:55:04.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 1 Processor -- start 2023-10-30T17:55:04.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 1 -- start start 2023-10-30T17:55:04.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 1 librados: setting wanted keys 2023-10-30T17:55:04.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 1 librados: calling monclient init 2023-10-30T17:55:04.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 10 monclient: init 2023-10-30T17:55:04.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:04.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:55:04.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b1c0b7c90 0x7f6b1c141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 10 monclient(hunting): picked mon.noname-a con 0x7f6b1c0b7c90 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:04.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b1c13eb60 0x7f6b1c1da180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 10 monclient(hunting): picked mon.noname-b con 0x7f6b1c13eb60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:04.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.635+0000 7f6b2433e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1c144f50 0x7f6b1c1d62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b2433e640 10 monclient(hunting): picked mon.noname-c con 0x7f6b1c144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:04.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b2433e640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:04.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b2433e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c134460 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b2433e640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:04.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b2433e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c1da6c0 con 0x7f6b1c13eb60 2023-10-30T17:55:04.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b2433e640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:04.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b2433e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c1d9c60 con 0x7f6b1c144f50 2023-10-30T17:55:04.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b2433e640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:04.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b2433e640 10 monclient(hunting): authenticate will time out at 2686.959082s 2023-10-30T17:55:04.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b220da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b1c0b7c90 0x7f6b1c141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:04.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b228db640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1c144f50 0x7f6b1c1d62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:04.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b218d9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b1c13eb60 0x7f6b1c1da180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:04.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b220da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b1c0b7c90 0x7f6b1c141bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40428/0 (socket says 172.21.15.170:40428) 2023-10-30T17:55:04.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b220da640 1 -- 172.21.15.170:0/2148875277 learned_addr learned my addr 172.21.15.170:0/2148875277 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:04.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b220da640 10 monclient(hunting): get_auth_request con 0x7f6b1c0b7c90 auth_method 0 2023-10-30T17:55:04.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b220da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:04.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b220da640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:04.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b220da640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:04.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b218d9640 10 monclient(hunting): get_auth_request con 0x7f6b1c13eb60 auth_method 0 2023-10-30T17:55:04.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b218d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:04.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b218d9640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:04.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b218d9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:04.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b228db640 10 monclient(hunting): get_auth_request con 0x7f6b1c144f50 auth_method 0 2023-10-30T17:55:04.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b228db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:04.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b228db640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:04.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.636+0000 7f6b228db640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:04.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.637+0000 7f6b220da640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:04.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.637+0000 7f6b220da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:04.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.637+0000 7f6b220da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:04.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.637+0000 7f6b218d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:04.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.637+0000 7f6b218d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:04.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.637+0000 7f6b218d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:04.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.637+0000 7f6b228db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:04.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.637+0000 7f6b228db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:04.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.637+0000 7f6b228db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:04.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.639+0000 7f6b220da640 10 monclient(hunting): handle_auth_done global_id 16005 payload 824 2023-10-30T17:55:04.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.639+0000 7f6b220da640 1 -- 172.21.15.170:0/2148875277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1c144f50 msgr2=0x7f6b1c1d62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.639+0000 7f6b220da640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1c144f50 0x7f6b1c1d62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:04.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.639+0000 7f6b220da640 1 -- 172.21.15.170:0/2148875277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b1c13eb60 msgr2=0x7f6b1c1da180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:04.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.639+0000 7f6b220da640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b1c13eb60 0x7f6b1c1da180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:04.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.639+0000 7f6b220da640 10 monclient: _finish_hunting 0 2023-10-30T17:55:04.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.639+0000 7f6b220da640 1 monclient: found mon.noname-a 2023-10-30T17:55:04.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.639+0000 7f6b220da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:04.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.639+0000 7f6b220da640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:04.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.639+0000 7f6b220da640 1 -- 172.21.15.170:0/2148875277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b18009a70 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.642+0000 7f6b220da640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b1c0b7c90 0x7f6b1c141bf0 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7f6b1400c500 tx=0x7f6b1400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:04.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.642+0000 7f6b0effd640 10 client.?.objecter ms_handle_connect 0x7f6b1c0b7c90 2023-10-30T17:55:04.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.642+0000 7f6b0effd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:04.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 1 -- 172.21.15.170:0/2148875277 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b14005ce0 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:04.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:04.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 10 monclient: dump: 2023-10-30T17:55:04.672 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:04.673 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:04.673 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.673 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.673 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:04.673 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.674 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:04.674 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:04.674 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:04.674 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 10 monclient: _finish_auth 0 2023-10-30T17:55:04.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 10 monclient: _check_auth_tickets 2023-10-30T17:55:04.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:04.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 1 -- 172.21.15.170:0/2148875277 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b14005e40 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b2433e640 5 monclient: authenticate success, global_id 16005 2023-10-30T17:55:04.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b2433e640 10 monclient: _renew_subs 2023-10-30T17:55:04.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b2433e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:04.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b2433e640 1 -- 172.21.15.170:0/2148875277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1c1d6810 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:04.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 1 -- 172.21.15.170:0/2148875277 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b14007330 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:04.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:04.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b0effd640 10 monclient: dump: 2023-10-30T17:55:04.677 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:04.677 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:04.677 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.677 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:04.678 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:04.678 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:04.678 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:04.678 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:04.678 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:04.679 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:04.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b208d7640 4 set_mon_vals no callback set 2023-10-30T17:55:04.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b2433e640 10 client.16005.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:04.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b2433e640 10 monclient: _renew_subs 2023-10-30T17:55:04.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b2433e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:04.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b2433e640 1 -- 172.21.15.170:0/2148875277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1c1d97d0 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.645+0000 7f6b2433e640 1 librados: init done 2023-10-30T17:55:04.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.646+0000 7f6b2433e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:04.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.646+0000 7f6b2433e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:04.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.646+0000 7f6b2433e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:04.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.646+0000 7f6b2433e640 1 -- 172.21.15.170:0/2148875277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1c1d9ab0 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.897+0000 7f6b0effd640 1 -- 172.21.15.170:0/2148875277 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6b14007490 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.898+0000 7f6b0effd640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f6b0006c850 0x7f6b0006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:04.900 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.898+0000 7f6b218d9640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f6b0006c850 0x7f6b0006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:04.900 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.898+0000 7f6b218d9640 10 monclient: get_auth_request con 0x7f6b0006c850 auth_method 0 2023-10-30T17:55:04.900 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.899+0000 7f6b218d9640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f6b0006c850 0x7f6b0006ed10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c142440 tx=0x7f6b10009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:04.901 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.899+0000 7f6b0effd640 10 client.16005.objecter ms_handle_connect 0x7f6b0006c850 2023-10-30T17:55:04.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.913+0000 7f6b0effd640 1 -- 172.21.15.170:0/2148875277 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f6b14007740 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.913+0000 7f6b0effd640 10 client.16005.objecter ms_dispatch 0x7f6b1c000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:04.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.913+0000 7f6b0effd640 3 client.16005.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:04.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.913+0000 7f6b0effd640 3 client.16005.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:04.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.913+0000 7f6b0effd640 20 client.16005.objecter dump_active .. 0 homeless 2023-10-30T17:55:04.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.913+0000 7f6b0effd640 1 -- 172.21.15.170:0/2148875277 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6b14054270 con 0x7f6b1c0b7c90 2023-10-30T17:55:04.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.913+0000 7f6b0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:04.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:04.913+0000 7f6b0effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:05.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.095+0000 7f6b2433e640 1 -- 172.21.15.170:0/2148875277 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b1c0b8070 con 0x7f6b0006c850 2023-10-30T17:55:05.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.095+0000 7f6b0effd640 1 -- 172.21.15.170:0/2148875277 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6b100014c0 con 0x7f6b0006c850 2023-10-30T17:55:05.097 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T17:55:05.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 10 librados: watch_flush enter 2023-10-30T17:55:05.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 10 librados: watch_flush exit 2023-10-30T17:55:05.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 20 client.16005.objecter shutdown clearing up homeless session... 2023-10-30T17:55:05.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 10 client.16005.objecter successfully canceled tick 2023-10-30T17:55:05.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 -- 172.21.15.170:0/2148875277 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f6b0006c850 msgr2=0x7f6b0006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f6b0006c850 0x7f6b0006ed10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c142440 tx=0x7f6b10009040 comp rx=0 tx=0).stop 2023-10-30T17:55:05.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 10 monclient: shutdown 2023-10-30T17:55:05.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:05.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 -- 172.21.15.170:0/2148875277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b1c0b7c90 msgr2=0x7f6b1c141bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b1c0b7c90 0x7f6b1c141bf0 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7f6b1400c500 tx=0x7f6b1400c9c0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 -- 172.21.15.170:0/2148875277 shutdown_connections 2023-10-30T17:55:05.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f6b0006c850 0x7f6b0006ed10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b1c13eb60 0x7f6b1c1da180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1c144f50 0x7f6b1c1d62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 --2- 172.21.15.170:0/2148875277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b1c0b7c90 0x7f6b1c141bf0 unknown :-1 s=CLOSED pgs=3367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 -- 172.21.15.170:0/2148875277 >> 172.21.15.170:0/2148875277 conn(0x7f6b1c071da0 msgr2=0x7f6b1c0b72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:05.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 -- 172.21.15.170:0/2148875277 shutdown_connections 2023-10-30T17:55:05.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 -- 172.21.15.170:0/2148875277 wait complete. 2023-10-30T17:55:05.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.098+0000 7f6b2433e640 1 librados: shutdown 2023-10-30T17:55:05.113 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T17:55:05.113 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T17:55:05.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.218+0000 7fad63a51640 10 monclient: get_monmap_and_config 2023-10-30T17:55:05.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.218+0000 7fad63a51640 10 monclient: build_initial_monmap 2023-10-30T17:55:05.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.218+0000 7fad63a51640 10 monclient: monmap: 2023-10-30T17:55:05.220 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:05.220 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:05.220 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:05.219280+0000 2023-10-30T17:55:05.221 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:05.219280+0000 2023-10-30T17:55:05.221 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:05.221 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.221 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:05.221 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:05.221 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:05.222 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:05.222 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:05.222 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.219+0000 7fad63a51640 1 Processor -- start 2023-10-30T17:55:05.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.219+0000 7fad63a51640 1 -- start start 2023-10-30T17:55:05.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.219+0000 7fad63a51640 10 monclient: init 2023-10-30T17:55:05.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:05.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-10-30T17:55:05.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 0x7fad5c0ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 10 monclient(hunting): picked mon.noname-b con 0x7fad5c0b4460 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:05.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad5c13eb10 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:05.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fad5c0b08c0 0x7fad5c0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 10 monclient(hunting): picked mon.noname-c con 0x7fad5c0b08c0 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:05.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c0aeda0 con 0x7fad5c13eb10 2023-10-30T17:55:05.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:05.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5c0b4840 con 0x7fad5c0b08c0 2023-10-30T17:55:05.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:05.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c0b04a0 con 0x7fad5c0b4460 2023-10-30T17:55:05.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:05.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.220+0000 7fad63a51640 10 monclient(hunting): authenticate will time out at 2687.543438s 2023-10-30T17:55:05.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.224+0000 7fad617ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 0x7fad5c0ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad617ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 0x7fad5c0ae860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:38926/0 (socket says 172.21.15.170:38926) 2023-10-30T17:55:05.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad617ed640 1 -- 172.21.15.170:0/3699451576 learned_addr learned my addr 172.21.15.170:0/3699451576 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:05.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad617ed640 10 monclient(hunting): get_auth_request con 0x7fad5c0b4460 auth_method 0 2023-10-30T17:55:05.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad617ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:05.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad617ed640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad617ed640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad61fee640 1 --2- 172.21.15.170:0/3699451576 >> v2:172.21.15.177:3300/0 conn(0x7fad5c0b08c0 0x7fad5c0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad61fee640 10 monclient(hunting): get_auth_request con 0x7fad5c0b08c0 auth_method 0 2023-10-30T17:55:05.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad61fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:05.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad61fee640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.225+0000 7fad61fee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.226+0000 7fad61fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:05.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.226+0000 7fad61fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:05.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.226+0000 7fad61fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:05.270 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad617ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:05.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad617ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:05.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad617ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:05.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad61fee640 10 monclient(hunting): handle_auth_done global_id 21454 payload 274 2023-10-30T17:55:05.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad53fff640 1 -- 172.21.15.170:0/3699451576 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489088423 0 0) 0x7fad5c0aeda0 con 0x7fad5c13eb10 2023-10-30T17:55:05.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad61fee640 1 -- 172.21.15.170:0/3699451576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 msgr2=0x7fad5c0ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:05.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad61fee640 1 --2- 172.21.15.170:0/3699451576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 0x7fad5c0ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad61fee640 1 -- 172.21.15.170:0/3699451576 >> v1:172.21.15.170:6789/0 conn(0x7fad5c13eb10 legacy=0x7fad5c13ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad61fee640 10 monclient: _finish_hunting 0 2023-10-30T17:55:05.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad61fee640 1 monclient: found mon.noname-c 2023-10-30T17:55:05.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad61fee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:05.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad61fee640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:55:05.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad61fee640 1 -- 172.21.15.170:0/3699451576 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c0af010 con 0x7fad5c0b08c0 2023-10-30T17:55:05.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.269+0000 7fad53fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T17:55:05.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.273+0000 7fad61fee640 1 --2- 172.21.15.170:0/3699451576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0b08c0 0x7fad5c0b0d50 secure :-1 s=READY pgs=5272 cs=0 l=1 rev1=1 crypto rx=0x7fad54009dc0 tx=0x7fad5402be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91de01b94510cd4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:05.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 1 -- 172.21.15.170:0/3699451576 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5402cbb0 con 0x7fad5c0b08c0 2023-10-30T17:55:05.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:05.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:05.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 10 monclient: dump: 2023-10-30T17:55:05.279 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:05.280 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:05.280 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.280 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.280 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:05.280 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.281 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:05.281 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:05.281 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:05.281 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 10 monclient: _finish_auth 0 2023-10-30T17:55:05.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 10 monclient: _check_auth_tickets 2023-10-30T17:55:05.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:05.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 1 -- 172.21.15.170:0/3699451576 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5402cd10 con 0x7fad5c0b08c0 2023-10-30T17:55:05.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad63a51640 5 monclient: authenticate success, global_id 21454 2023-10-30T17:55:05.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad63a51640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:05.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:05.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad63a51640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:05.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 1 -- 172.21.15.170:0/3699451576 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54034660 con 0x7fad5c0b08c0 2023-10-30T17:55:05.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad63a51640 4 set_mon_vals no callback set 2023-10-30T17:55:05.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:05.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:05.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.276+0000 7fad53fff640 10 monclient: dump: 2023-10-30T17:55:05.284 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:05.284 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:05.284 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.284 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.285 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:05.285 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.285 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:05.285 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:05.285 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:05.286 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.277+0000 7fad63a51640 10 monclient: shutdown 2023-10-30T17:55:05.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.277+0000 7fad63a51640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:05.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.277+0000 7fad63a51640 1 -- 172.21.15.170:0/3699451576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0b08c0 msgr2=0x7fad5c0b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.277+0000 7fad63a51640 1 --2- 172.21.15.170:0/3699451576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0b08c0 0x7fad5c0b0d50 secure :-1 s=READY pgs=5272 cs=0 l=1 rev1=1 crypto rx=0x7fad54009dc0 tx=0x7fad5402be00 comp rx=0 tx=0).stop 2023-10-30T17:55:05.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.277+0000 7fad63a51640 1 -- 172.21.15.170:0/3699451576 shutdown_connections 2023-10-30T17:55:05.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.277+0000 7fad63a51640 1 --2- 172.21.15.170:0/3699451576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0b08c0 0x7fad5c0b0d50 unknown :-1 s=CLOSED pgs=5272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.277+0000 7fad63a51640 1 --2- 172.21.15.170:0/3699451576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 0x7fad5c0ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.277+0000 7fad63a51640 1 -- 172.21.15.170:0/3699451576 >> 172.21.15.170:0/3699451576 conn(0x7fad5c071b90 msgr2=0x7fad5c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:05.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.277+0000 7fad63a51640 1 -- 172.21.15.170:0/3699451576 shutdown_connections 2023-10-30T17:55:05.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.277+0000 7fad63a51640 1 -- 172.21.15.170:0/3699451576 wait complete. 2023-10-30T17:55:05.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.278+0000 7fad63a51640 10 monclient: build_initial_monmap 2023-10-30T17:55:05.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.278+0000 7fad63a51640 10 monclient: monmap: 2023-10-30T17:55:05.292 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:05.292 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:05.292 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:05.293 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:05.293 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:05.293 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.293 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:05.293 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:05.294 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:05.294 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.278+0000 7fad63a51640 1 librados: starting msgr at 2023-10-30T17:55:05.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.278+0000 7fad63a51640 1 librados: starting objecter 2023-10-30T17:55:05.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.278+0000 7fad63a51640 1 Processor -- start 2023-10-30T17:55:05.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.278+0000 7fad63a51640 1 -- start start 2023-10-30T17:55:05.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.278+0000 7fad63a51640 1 librados: setting wanted keys 2023-10-30T17:55:05.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.278+0000 7fad63a51640 1 librados: calling monclient init 2023-10-30T17:55:05.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.278+0000 7fad63a51640 10 monclient: init 2023-10-30T17:55:05.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:05.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T17:55:05.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0b08c0 0x7fad5c1423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 10 monclient(hunting): picked mon.noname-b con 0x7fad5c0b08c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:05.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 0x7fad5c1da1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 10 monclient(hunting): picked mon.noname-c con 0x7fad5c0b4460 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:05.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad5c13eb10 0x7fad5c139120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 10 monclient(hunting): picked mon.noname-a con 0x7fad5c13eb10 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:05.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:05.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c1413b0 con 0x7fad5c13eb10 2023-10-30T17:55:05.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:05.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c1d6f00 con 0x7fad5c0b08c0 2023-10-30T17:55:05.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:05.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c139660 con 0x7fad5c0b4460 2023-10-30T17:55:05.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:05.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.279+0000 7fad63a51640 10 monclient(hunting): authenticate will time out at 2687.602592s 2023-10-30T17:55:05.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad60fec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 0x7fad5c1da1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad60fec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 0x7fad5c1da1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:38940/0 (socket says 172.21.15.170:38940) 2023-10-30T17:55:05.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad60fec640 1 -- 172.21.15.170:0/1062513746 learned_addr learned my addr 172.21.15.170:0/1062513746 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:05.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad60fec640 10 monclient(hunting): get_auth_request con 0x7fad5c0b4460 auth_method 0 2023-10-30T17:55:05.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad60fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:05.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad60fec640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad60fec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad617ed640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0b08c0 0x7fad5c1423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad617ed640 10 monclient(hunting): get_auth_request con 0x7fad5c0b08c0 auth_method 0 2023-10-30T17:55:05.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad617ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:05.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad617ed640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.280+0000 7fad617ed640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.281+0000 7fad60fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:05.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.281+0000 7fad60fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:05.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.281+0000 7fad60fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:05.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.281+0000 7fad617ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:05.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.281+0000 7fad617ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:05.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.281+0000 7fad617ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:05.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad61fee640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad5c13eb10 0x7fad5c139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad61fee640 10 monclient(hunting): get_auth_request con 0x7fad5c13eb10 auth_method 0 2023-10-30T17:55:05.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad61fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:05.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad61fee640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad61fee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad617ed640 10 monclient(hunting): handle_auth_done global_id 21457 payload 824 2023-10-30T17:55:05.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad617ed640 1 -- 172.21.15.170:0/1062513746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 msgr2=0x7fad5c1da1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad617ed640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad5c0b4460 0x7fad5c1da1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad617ed640 1 -- 172.21.15.170:0/1062513746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad5c13eb10 msgr2=0x7fad5c139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad617ed640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad5c13eb10 0x7fad5c139120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad617ed640 10 monclient: _finish_hunting 0 2023-10-30T17:55:05.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad617ed640 1 monclient: found mon.noname-b 2023-10-30T17:55:05.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad617ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:05.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad617ed640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:05.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.283+0000 7fad617ed640 1 -- 172.21.15.170:0/1062513746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad54009a70 con 0x7fad5c0b08c0 2023-10-30T17:55:05.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.285+0000 7fad617ed640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0b08c0 0x7fad5c1423d0 secure :-1 s=READY pgs=5273 cs=0 l=1 rev1=1 crypto rx=0x7fad5800e850 tx=0x7fad5800ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:05.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.285+0000 7fad51ffb640 10 client.?.objecter ms_handle_connect 0x7fad5c0b08c0 2023-10-30T17:55:05.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.285+0000 7fad51ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:05.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 1 -- 172.21.15.170:0/1062513746 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5800c940 con 0x7fad5c0b08c0 2023-10-30T17:55:05.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:05.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:05.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 10 monclient: dump: 2023-10-30T17:55:05.308 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:05.308 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:05.308 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.308 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.308 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:05.308 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.309 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:05.309 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:05.309 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:05.309 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 10 monclient: _finish_auth 0 2023-10-30T17:55:05.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:55:05.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:05.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 1 -- 172.21.15.170:0/1062513746 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5800caa0 con 0x7fad5c0b08c0 2023-10-30T17:55:05.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:05.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 1 -- 172.21.15.170:0/1062513746 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5800ce10 con 0x7fad5c0b08c0 2023-10-30T17:55:05.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:05.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:05.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad51ffb640 10 monclient: dump: 2023-10-30T17:55:05.311 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:05.311 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:05.312 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.312 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.312 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:05.312 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.312 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:05.313 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:05.313 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:05.313 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad63a51640 5 monclient: authenticate success, global_id 21457 2023-10-30T17:55:05.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad63a51640 10 monclient: _renew_subs 2023-10-30T17:55:05.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad63a51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:05.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c139a70 con 0x7fad5c0b08c0 2023-10-30T17:55:05.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad537fe640 4 set_mon_vals no callback set 2023-10-30T17:55:05.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad63a51640 10 client.21457.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:05.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad63a51640 10 monclient: _renew_subs 2023-10-30T17:55:05.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad63a51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:05.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c1d7600 con 0x7fad5c0b08c0 2023-10-30T17:55:05.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.289+0000 7fad63a51640 1 librados: init done 2023-10-30T17:55:05.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.290+0000 7fad63a51640 10 client.21457.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:05.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.290+0000 7fad63a51640 15 client.21457.objecter _session_command_op_assign -1 1 2023-10-30T17:55:05.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.290+0000 7fad63a51640 20 client.21457.objecter _get_session osd=-1 returning homeless 2023-10-30T17:55:05.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.290+0000 7fad63a51640 10 client.21457.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:05.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.290+0000 7fad63a51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:05.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.290+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fad5c1d6d90 con 0x7fad5c0b08c0 2023-10-30T17:55:05.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.310+0000 7fad51ffb640 1 -- 172.21.15.170:0/1062513746 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fad58013470 con 0x7fad5c0b08c0 2023-10-30T17:55:05.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.310+0000 7fad51ffb640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fad4006c800 0x7fad4006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.310+0000 7fad60fec640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fad4006c800 0x7fad4006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.310+0000 7fad60fec640 10 monclient: get_auth_request con 0x7fad4006c800 auth_method 0 2023-10-30T17:55:05.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.311+0000 7fad60fec640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fad4006c800 0x7fad4006ecc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fad5c13e650 tx=0x7fad4c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:05.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.311+0000 7fad51ffb640 10 client.21457.objecter ms_handle_connect 0x7fad4006c800 2023-10-30T17:55:05.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.312+0000 7fad51ffb640 1 -- 172.21.15.170:0/1062513746 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fad58058900 con 0x7fad5c0b08c0 2023-10-30T17:55:05.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.312+0000 7fad51ffb640 10 client.21457.objecter ms_dispatch 0x7fad5c000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:05.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.312+0000 7fad51ffb640 3 client.21457.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:05.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 3 client.21457.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:05.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 10 client.21457.objecter checking command 1 2023-10-30T17:55:05.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] conn(0x7fad40073f80 0x7fad40076420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 20 client.21457.objecter _get_session s=0x7fad400738b0 osd=1 3 2023-10-30T17:55:05.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 20 client.21457.objecter put_session s=0x7fad400738b0 osd=1 3 2023-10-30T17:55:05.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 15 client.21457.objecter _session_command_op_remove -1 1 2023-10-30T17:55:05.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 20 client.21457.objecter _get_session s=0x7fad400738b0 osd=1 3 2023-10-30T17:55:05.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 20 client.21457.objecter get_session s=0x7fad400738b0 osd=1 3 2023-10-30T17:55:05.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 15 client.21457.objecter _session_command_op_assign 1 1 2023-10-30T17:55:05.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 20 client.21457.objecter put_session s=0x7fad400738b0 osd=1 4 2023-10-30T17:55:05.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 10 client.21457.objecter _send_command 1 2023-10-30T17:55:05.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 1 -- 172.21.15.170:0/1062513746 --> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad40076af0 con 0x7fad40073f80 2023-10-30T17:55:05.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 20 client.21457.objecter dump_active .. 0 homeless 2023-10-30T17:55:05.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 1 -- 172.21.15.170:0/1062513746 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=914) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad58054c40 con 0x7fad5c0b08c0 2023-10-30T17:55:05.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad51ffb640 10 monclient: handle_get_version_reply finishing 1 version 914 2023-10-30T17:55:05.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.313+0000 7fad61fee640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] conn(0x7fad40073f80 0x7fad40076420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.314+0000 7fad61fee640 10 monclient: get_auth_request con 0x7fad40073f80 auth_method 0 2023-10-30T17:55:05.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.316+0000 7fad51ffb640 10 client.21457.objecter ms_handle_connect 0x7fad40073f80 2023-10-30T17:55:05.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.316+0000 7fad61fee640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] conn(0x7fad40073f80 0x7fad40076420 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:05.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.339+0000 7fad51ffb640 1 -- 172.21.15.170:0/1062513746 <== osd.1 v2:172.21.15.170:6824/527201623 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fad40076af0 con 0x7fad40073f80 2023-10-30T17:55:05.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.339+0000 7fad51ffb640 10 client.21457.objecter ms_dispatch 0x7fad5c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:55:05.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.339+0000 7fad51ffb640 10 client.21457.objecter _finish_command 1 = system:0 2023-10-30T17:55:05.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.339+0000 7fad51ffb640 20 client.21457.objecter put_session s=0x7fad400738b0 osd=1 4 2023-10-30T17:55:05.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.339+0000 7fad51ffb640 15 client.21457.objecter _session_command_op_remove 1 1 2023-10-30T17:55:05.355 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 10 client.21457.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T17:55:05.355 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 15 client.21457.objecter _session_command_op_assign -1 2 2023-10-30T17:55:05.355 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 20 client.21457.objecter _get_session s=0x7fad400738b0 osd=1 3 2023-10-30T17:55:05.355 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 20 client.21457.objecter put_session s=0x7fad400738b0 osd=1 3 2023-10-30T17:55:05.356 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 20 client.21457.objecter _get_session s=0x7fad400738b0 osd=1 3 2023-10-30T17:55:05.356 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 15 client.21457.objecter _session_command_op_remove -1 2 2023-10-30T17:55:05.356 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 20 client.21457.objecter get_session s=0x7fad400738b0 osd=1 3 2023-10-30T17:55:05.356 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 15 client.21457.objecter _session_command_op_assign 1 2 2023-10-30T17:55:05.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 20 client.21457.objecter put_session s=0x7fad400738b0 osd=1 4 2023-10-30T17:55:05.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 10 client.21457.objecter _send_command 2 2023-10-30T17:55:05.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.354+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 --> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fad5c0b3a40 con 0x7fad40073f80 2023-10-30T17:55:05.368 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad51ffb640 1 -- 172.21.15.170:0/1062513746 <== osd.1 v2:172.21.15.170:6824/527201623 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fad540344c0 con 0x7fad40073f80 2023-10-30T17:55:05.368 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad51ffb640 10 client.21457.objecter ms_dispatch 0x7fad5c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:55:05.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad51ffb640 10 client.21457.objecter _finish_command 2 = system:0 2023-10-30T17:55:05.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad51ffb640 20 client.21457.objecter put_session s=0x7fad400738b0 osd=1 4 2023-10-30T17:55:05.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad51ffb640 15 client.21457.objecter _session_command_op_remove 1 2 2023-10-30T17:55:05.369 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-30T17:55:05.370 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T17:55:05.370 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-30T17:55:05.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad63a51640 10 librados: watch_flush enter 2023-10-30T17:55:05.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad63a51640 10 librados: watch_flush exit 2023-10-30T17:55:05.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad63a51640 10 client.21457.objecter close_session for osd.1 2023-10-30T17:55:05.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 >> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] conn(0x7fad40073f80 msgr2=0x7fad40076420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad63a51640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.170:6824/527201623,v1:172.21.15.170:6825/527201623] conn(0x7fad40073f80 0x7fad40076420 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad63a51640 20 client.21457.objecter put_session s=0x7fad400738b0 osd=1 1 2023-10-30T17:55:05.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad63a51640 20 client.21457.objecter shutdown clearing up homeless session... 2023-10-30T17:55:05.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.367+0000 7fad63a51640 10 client.21457.objecter successfully canceled tick 2023-10-30T17:55:05.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fad4006c800 msgr2=0x7fad4006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fad4006c800 0x7fad4006ecc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fad5c13e650 tx=0x7fad4c009040 comp rx=0 tx=0).stop 2023-10-30T17:55:05.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 10 monclient: shutdown 2023-10-30T17:55:05.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:05.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0b08c0 msgr2=0x7fad5c1423d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 1 --2- 172.21.15.170:0/1062513746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5c0b08c0 0x7fad5c1423d0 secure :-1 s=READY pgs=5273 cs=0 l=1 rev1=1 crypto rx=0x7fad5800e850 tx=0x7fad5800ed10 comp rx=0 tx=0).stop 2023-10-30T17:55:05.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad61fee640 1 -- 172.21.15.170:0/1062513746 reap_dead start 2023-10-30T17:55:05.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 shutdown_connections 2023-10-30T17:55:05.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 >> 172.21.15.170:0/1062513746 conn(0x7fad5c071b90 msgr2=0x7fad5c1cc730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:05.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 shutdown_connections 2023-10-30T17:55:05.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 1 -- 172.21.15.170:0/1062513746 wait complete. 2023-10-30T17:55:05.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.368+0000 7fad63a51640 1 librados: shutdown 2023-10-30T17:55:05.380 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T17:55:05.480 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.479+0000 7f75b4990640 10 monclient: get_monmap_and_config 2023-10-30T17:55:05.480 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.479+0000 7f75b4990640 10 monclient: build_initial_monmap 2023-10-30T17:55:05.480 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.479+0000 7f75b4990640 10 monclient: monmap: 2023-10-30T17:55:05.481 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:05.481 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:05.481 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:05.480011+0000 2023-10-30T17:55:05.481 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:05.480011+0000 2023-10-30T17:55:05.482 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:05.482 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.482 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:05.482 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:05.483 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:05.483 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:05.483 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:05.483 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.479+0000 7f75b4990640 1 Processor -- start 2023-10-30T17:55:05.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.479+0000 7f75b4990640 1 -- start start 2023-10-30T17:55:05.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.479+0000 7f75b4990640 10 monclient: init 2023-10-30T17:55:05.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:05.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-10-30T17:55:05.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f75b00b7c90 0x7f75b00a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 10 monclient(hunting): picked mon.noname-a con 0x7f75b00b7c90 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:05.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b013eb60 0x7f75b013efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 10 monclient(hunting): picked mon.noname-b con 0x7f75b013eb60 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:05.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75b0144f50 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:05.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b0145ad0 con 0x7f75b0144f50 2023-10-30T17:55:05.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:05.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b00a9500 con 0x7f75b00b7c90 2023-10-30T17:55:05.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:05.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b00b8070 con 0x7f75b013eb60 2023-10-30T17:55:05.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:05.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.480+0000 7f75b4990640 10 monclient(hunting): authenticate will time out at 2687.803593s 2023-10-30T17:55:05.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75ae59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f75b00b7c90 0x7f75b00a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75aed9d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f75b0144f50 0x7f75b01453c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:45920/0 (socket says 172.21.15.170:45920) 2023-10-30T17:55:05.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75aed9d640 1 -- 172.21.15.170:0/2088130034 learned_addr learned my addr 172.21.15.170:0/2088130034 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:05.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75add9b640 1 --2- 172.21.15.170:0/2088130034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b013eb60 0x7f75b013efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75ae59c640 10 monclient(hunting): get_auth_request con 0x7f75b00b7c90 auth_method 0 2023-10-30T17:55:05.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:05.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75ae59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75ae59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.492 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75add9b640 10 monclient(hunting): get_auth_request con 0x7f75b013eb60 auth_method 0 2023-10-30T17:55:05.492 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:05.492 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75add9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.492 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.481+0000 7f75add9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.482+0000 7f75ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:05.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.482+0000 7f75ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:05.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.482+0000 7f75ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:05.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.482+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:05.494 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.482+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:05.494 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.482+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:05.494 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75ad59a640 1 -- 172.21.15.170:0/2088130034 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896170514 0 0) 0x7f75b0145ad0 con 0x7f75b0144f50 2023-10-30T17:55:05.494 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75ad59a640 10 monclient(hunting): _negotiate 2023-10-30T17:55:05.495 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75ad59a640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.495 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75ad59a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.495 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75ad59a640 10 monclient(hunting): my global_id is 21460 2023-10-30T17:55:05.495 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75ad59a640 1 -- 172.21.15.170:0/2088130034 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7598003520 con 0x7f75b0144f50 2023-10-30T17:55:05.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75aed9d640 1 -- 172.21.15.170:0/2088130034 >> v1:172.21.15.177:6789/0 conn(0x7f75b0144f50 legacy=0x7f75b01453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2023-10-30T17:55:05.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75aed9d640 1 -- 172.21.15.170:0/2088130034 >> v1:172.21.15.177:6789/0 conn(0x7f75b0144f50 legacy=0x7f75b01453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T17:55:05.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75aed9d640 1 --1- 172.21.15.170:0/2088130034 >> v1:172.21.15.177:6789/0 conn(0x7f75b0144f50 0x7f75b01453c0 :-1 s=OPENED pgs=5274 cs=1 l=1).handle_message read tag failed 2023-10-30T17:55:05.496 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75aed9d640 1 --1- 172.21.15.170:0/2088130034 >> v1:172.21.15.177:6789/0 conn(0x7f75b0144f50 0x7f75b01453c0 :-1 s=OPENED pgs=5274 cs=1 l=1).fault on lossy channel, failing 2023-10-30T17:55:05.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.483+0000 7f75ad59a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.177:6789/0 2023-10-30T17:55:05.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.484+0000 7f75ae59c640 10 monclient(hunting): handle_auth_done global_id 16014 payload 274 2023-10-30T17:55:05.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.484+0000 7f75ae59c640 1 -- 172.21.15.170:0/2088130034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b013eb60 msgr2=0x7f75b013efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:05.498 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.484+0000 7f75ae59c640 1 --2- 172.21.15.170:0/2088130034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b013eb60 0x7f75b013efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.498 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.484+0000 7f75ae59c640 1 -- 172.21.15.170:0/2088130034 >> v1:172.21.15.177:6789/0 conn(0x7f75b0144f50 legacy=0x7f75b01453c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T17:55:05.498 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.484+0000 7f75ae59c640 10 monclient: _finish_hunting 0 2023-10-30T17:55:05.498 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.484+0000 7f75ae59c640 1 monclient: found mon.noname-a 2023-10-30T17:55:05.499 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.484+0000 7f75ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:05.499 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.484+0000 7f75ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:05.499 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.484+0000 7f75ae59c640 1 -- 172.21.15.170:0/2088130034 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b00aa740 con 0x7f75b00b7c90 2023-10-30T17:55:05.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.486+0000 7f75ae59c640 1 --2- 172.21.15.170:0/2088130034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 0x7f75b00a8b80 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f75b00b8280 tx=0x7f75a002b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e83358d1677701 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:05.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 1 -- 172.21.15.170:0/2088130034 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0004230 con 0x7f75b00b7c90 2023-10-30T17:55:05.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:05.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:05.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 10 monclient: dump: 2023-10-30T17:55:05.508 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:05.509 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:05.509 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.509 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.509 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:05.510 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.510 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:05.510 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:05.511 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:05.511 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 10 monclient: _finish_auth 0 2023-10-30T17:55:05.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 10 monclient: _check_auth_tickets 2023-10-30T17:55:05.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:05.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 1 -- 172.21.15.170:0/2088130034 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a002e050 con 0x7f75b00b7c90 2023-10-30T17:55:05.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75b4990640 5 monclient: authenticate success, global_id 16014 2023-10-30T17:55:05.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75b4990640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:05.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:05.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75b4990640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:05.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 1 -- 172.21.15.170:0/2088130034 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0003610 con 0x7f75b00b7c90 2023-10-30T17:55:05.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75b4990640 4 set_mon_vals no callback set 2023-10-30T17:55:05.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:05.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:05.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75ad59a640 10 monclient: dump: 2023-10-30T17:55:05.514 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:05.514 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:05.514 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.514 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.514 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:05.515 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.515 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:05.515 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:05.515 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:05.515 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.515 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75b4990640 10 monclient: shutdown 2023-10-30T17:55:05.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75b4990640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:05.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75b4990640 1 -- 172.21.15.170:0/2088130034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 msgr2=0x7f75b00a8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.490+0000 7f75b4990640 1 --2- 172.21.15.170:0/2088130034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 0x7f75b00a8b80 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f75b00b8280 tx=0x7f75a002b760 comp rx=0 tx=0).stop 2023-10-30T17:55:05.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.491+0000 7f75b4990640 1 -- 172.21.15.170:0/2088130034 shutdown_connections 2023-10-30T17:55:05.516 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.491+0000 7f75b4990640 1 --2- 172.21.15.170:0/2088130034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b013eb60 0x7f75b013efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.491+0000 7f75b4990640 1 --2- 172.21.15.170:0/2088130034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 0x7f75b00a8b80 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.491+0000 7f75b4990640 1 -- 172.21.15.170:0/2088130034 >> 172.21.15.170:0/2088130034 conn(0x7f75b0071da0 msgr2=0x7f75b00b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:05.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.491+0000 7f75b4990640 1 -- 172.21.15.170:0/2088130034 shutdown_connections 2023-10-30T17:55:05.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.491+0000 7f75b4990640 1 -- 172.21.15.170:0/2088130034 wait complete. 2023-10-30T17:55:05.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.491+0000 7f75b4990640 10 monclient: build_initial_monmap 2023-10-30T17:55:05.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.491+0000 7f75b4990640 10 monclient: monmap: 2023-10-30T17:55:05.518 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:05.518 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:05.518 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:05.518 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:05.519 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:05.519 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.519 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:05.519 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:05.519 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:05.520 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.492+0000 7f75b4990640 1 librados: starting msgr at 2023-10-30T17:55:05.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.492+0000 7f75b4990640 1 librados: starting objecter 2023-10-30T17:55:05.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.492+0000 7f75b4990640 1 Processor -- start 2023-10-30T17:55:05.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.492+0000 7f75b4990640 1 -- start start 2023-10-30T17:55:05.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.492+0000 7f75b4990640 1 librados: setting wanted keys 2023-10-30T17:55:05.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.492+0000 7f75b4990640 1 librados: calling monclient init 2023-10-30T17:55:05.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.492+0000 7f75b4990640 10 monclient: init 2023-10-30T17:55:05.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.492+0000 7f75b4990640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:05.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.492+0000 7f75b4990640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:55:05.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 0x7f75b0141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 10 monclient(hunting): picked mon.noname-a con 0x7f75b00b7c90 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:05.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b013eb60 0x7f75b01da5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 10 monclient(hunting): picked mon.noname-b con 0x7f75b013eb60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:05.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b0144f50 0x7f75b00ab0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 10 monclient(hunting): picked mon.noname-c con 0x7f75b0144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:05.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:05.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b00abdb0 con 0x7f75b00b7c90 2023-10-30T17:55:05.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:05.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b01d9dc0 con 0x7f75b013eb60 2023-10-30T17:55:05.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:05.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b01da0a0 con 0x7f75b0144f50 2023-10-30T17:55:05.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:05.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75b4990640 10 monclient(hunting): authenticate will time out at 2687.816137s 2023-10-30T17:55:05.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75ae59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 0x7f75b0141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75add9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b013eb60 0x7f75b01da5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75aed9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b0144f50 0x7f75b00ab0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75ae59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 0x7f75b0141a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40446/0 (socket says 172.21.15.170:40446) 2023-10-30T17:55:05.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75add9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b013eb60 0x7f75b01da5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:49526/0 (socket says 172.21.15.170:49526) 2023-10-30T17:55:05.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75ae59c640 1 -- 172.21.15.170:0/3943764459 learned_addr learned my addr 172.21.15.170:0/3943764459 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:05.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75ae59c640 10 monclient(hunting): get_auth_request con 0x7f75b00b7c90 auth_method 0 2023-10-30T17:55:05.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:05.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75ae59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75ae59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75add9b640 10 monclient(hunting): get_auth_request con 0x7f75b013eb60 auth_method 0 2023-10-30T17:55:05.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:05.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75add9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.493+0000 7f75add9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75aed9d640 10 monclient(hunting): get_auth_request con 0x7f75b0144f50 auth_method 0 2023-10-30T17:55:05.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:05.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75aed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:05.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75aed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:05.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:05.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:05.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:05.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:05.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:05.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:05.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:05.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:05.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.494+0000 7f75aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:05.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.497+0000 7f75ae59c640 10 monclient(hunting): handle_auth_done global_id 16017 payload 824 2023-10-30T17:55:05.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.497+0000 7f75ae59c640 1 -- 172.21.15.170:0/3943764459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b0144f50 msgr2=0x7f75b00ab0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.497+0000 7f75ae59c640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b0144f50 0x7f75b00ab0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.497+0000 7f75ae59c640 1 -- 172.21.15.170:0/3943764459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b013eb60 msgr2=0x7f75b01da5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.497+0000 7f75ae59c640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b013eb60 0x7f75b01da5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.497+0000 7f75ae59c640 10 monclient: _finish_hunting 0 2023-10-30T17:55:05.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.497+0000 7f75ae59c640 1 monclient: found mon.noname-a 2023-10-30T17:55:05.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.497+0000 7f75ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:05.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.497+0000 7f75ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:05.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.497+0000 7f75ae59c640 1 -- 172.21.15.170:0/3943764459 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0009400 con 0x7f75b00b7c90 2023-10-30T17:55:05.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.499+0000 7f75ae59c640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 0x7f75b0141a00 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7f75a0035040 tx=0x7f75a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:05.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.499+0000 7f759effd640 10 client.?.objecter ms_handle_connect 0x7f75b00b7c90 2023-10-30T17:55:05.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.499+0000 7f759effd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:05.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 1 -- 172.21.15.170:0/3943764459 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0003ea0 con 0x7f75b00b7c90 2023-10-30T17:55:05.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:05.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:05.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 10 monclient: dump: 2023-10-30T17:55:05.536 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:05.536 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:05.536 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.536 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.537 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:05.537 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.537 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:05.537 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:05.538 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:05.538 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 10 monclient: _finish_auth 0 2023-10-30T17:55:05.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 10 monclient: _check_auth_tickets 2023-10-30T17:55:05.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:05.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 1 -- 172.21.15.170:0/3943764459 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a0030730 con 0x7f75b00b7c90 2023-10-30T17:55:05.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:05.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 1 -- 172.21.15.170:0/3943764459 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0030ae0 con 0x7f75b00b7c90 2023-10-30T17:55:05.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:05.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:05.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f759effd640 10 monclient: dump: 2023-10-30T17:55:05.541 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:05.541 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:05.541 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.542 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:05.542 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:05.542 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:05.542 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:05.543 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:05.543 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:05.543 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:05.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f75b4990640 5 monclient: authenticate success, global_id 16017 2023-10-30T17:55:05.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f75b4990640 10 monclient: _renew_subs 2023-10-30T17:55:05.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f75b4990640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:05.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f75acd99640 4 set_mon_vals no callback set 2023-10-30T17:55:05.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f75b4990640 1 -- 172.21.15.170:0/3943764459 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b01438a0 con 0x7f75b00b7c90 2023-10-30T17:55:05.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f75b4990640 10 client.16017.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:05.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f75b4990640 10 monclient: _renew_subs 2023-10-30T17:55:05.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f75b4990640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:05.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f75b4990640 1 -- 172.21.15.170:0/3943764459 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b01dab00 con 0x7f75b00b7c90 2023-10-30T17:55:05.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.502+0000 7f75b4990640 1 librados: init done 2023-10-30T17:55:05.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.503+0000 7f75b4990640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:05.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.503+0000 7f75b4990640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:05.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.503+0000 7f75b4990640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:05.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.503+0000 7f75b4990640 1 -- 172.21.15.170:0/3943764459 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b01dacf0 con 0x7f75b00b7c90 2023-10-30T17:55:05.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.747+0000 7f759effd640 1 -- 172.21.15.170:0/3943764459 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f75a0039050 con 0x7f75b00b7c90 2023-10-30T17:55:05.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.747+0000 7f759effd640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f759006c850 0x7f759006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:05.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.748+0000 7f75add9b640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f759006c850 0x7f759006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:05.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.748+0000 7f75add9b640 10 monclient: get_auth_request con 0x7f759006c850 auth_method 0 2023-10-30T17:55:05.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.748+0000 7f75add9b640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f759006c850 0x7f759006ed10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f75b01d4670 tx=0x7f75a4009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:05.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.748+0000 7f759effd640 10 client.16017.objecter ms_handle_connect 0x7f759006c850 2023-10-30T17:55:05.762 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.761+0000 7f759effd640 1 -- 172.21.15.170:0/3943764459 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f75a00044b0 con 0x7f75b00b7c90 2023-10-30T17:55:05.762 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.761+0000 7f759effd640 10 client.16017.objecter ms_dispatch 0x7f75b0000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:05.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.761+0000 7f759effd640 3 client.16017.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:05.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.761+0000 7f759effd640 3 client.16017.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:05.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.761+0000 7f759effd640 20 client.16017.objecter dump_active .. 0 homeless 2023-10-30T17:55:05.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.761+0000 7f759effd640 1 -- 172.21.15.170:0/3943764459 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f75a0076580 con 0x7f75b00b7c90 2023-10-30T17:55:05.763 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.761+0000 7f759effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:05.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.761+0000 7f759effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:05.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.935+0000 7f75b4990640 1 -- 172.21.15.170:0/3943764459 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f75b00a90c0 con 0x7f759006c850 2023-10-30T17:55:05.937 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.936+0000 7f759effd640 1 -- 172.21.15.170:0/3943764459 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f75a4011a30 con 0x7f759006c850 2023-10-30T17:55:05.938 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T17:55:05.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 10 librados: watch_flush enter 2023-10-30T17:55:05.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 10 librados: watch_flush exit 2023-10-30T17:55:05.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 20 client.16017.objecter shutdown clearing up homeless session... 2023-10-30T17:55:05.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 10 client.16017.objecter successfully canceled tick 2023-10-30T17:55:05.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 -- 172.21.15.170:0/3943764459 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f759006c850 msgr2=0x7f759006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f759006c850 0x7f759006ed10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f75b01d4670 tx=0x7f75a4009190 comp rx=0 tx=0).stop 2023-10-30T17:55:05.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 10 monclient: shutdown 2023-10-30T17:55:05.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:05.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 -- 172.21.15.170:0/3943764459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 msgr2=0x7f75b0141a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:05.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 0x7f75b0141a00 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7f75a0035040 tx=0x7f75a00046c0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 -- 172.21.15.170:0/3943764459 shutdown_connections 2023-10-30T17:55:05.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f759006c850 0x7f759006ed10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b013eb60 0x7f75b01da5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b0144f50 0x7f75b00ab0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 --2- 172.21.15.170:0/3943764459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75b00b7c90 0x7f75b0141a00 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:05.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 -- 172.21.15.170:0/3943764459 >> 172.21.15.170:0/3943764459 conn(0x7f75b0071da0 msgr2=0x7f75b00b72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:05.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 -- 172.21.15.170:0/3943764459 shutdown_connections 2023-10-30T17:55:05.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 -- 172.21.15.170:0/3943764459 wait complete. 2023-10-30T17:55:05.943 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:05.938+0000 7f75b4990640 1 librados: shutdown 2023-10-30T17:55:05.951 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T17:55:05.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T17:55:06.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.053+0000 7fe54027c640 10 monclient: get_monmap_and_config 2023-10-30T17:55:06.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.053+0000 7fe54027c640 10 monclient: build_initial_monmap 2023-10-30T17:55:06.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.053+0000 7fe54027c640 10 monclient: monmap: 2023-10-30T17:55:06.055 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:06.055 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.055 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:06.053978+0000 2023-10-30T17:55:06.055 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:06.053978+0000 2023-10-30T17:55:06.055 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:06.055 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.056 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:06.056 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:06.056 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:06.056 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:06.057 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:06.057 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.053+0000 7fe54027c640 1 Processor -- start 2023-10-30T17:55:06.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.053+0000 7fe54027c640 1 -- start start 2023-10-30T17:55:06.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.053+0000 7fe54027c640 10 monclient: init 2023-10-30T17:55:06.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:06.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-10-30T17:55:06.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5380b4460 0x7fe5380ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 10 monclient(hunting): picked mon.noname-b con 0x7fe5380b4460 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:06.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe53813eb10 0x7fe53813ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 10 monclient(hunting): picked mon.noname-c con 0x7fe53813eb10 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:06.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5380b08c0 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:06.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5380aeda0 con 0x7fe5380b08c0 2023-10-30T17:55:06.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5380b4840 con 0x7fe53813eb10 2023-10-30T17:55:06.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5380b04a0 con 0x7fe5380b4460 2023-10-30T17:55:06.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:06.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe54027c640 10 monclient(hunting): authenticate will time out at 2688.377501s 2023-10-30T17:55:06.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe53e819640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fe5380b08c0 0x7fe5380b0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59462/0 (socket says 172.21.15.170:59462) 2023-10-30T17:55:06.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.054+0000 7fe53e819640 1 -- 172.21.15.170:0/305469561 learned_addr learned my addr 172.21.15.170:0/305469561 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:06.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53e018640 1 --2- 172.21.15.170:0/305469561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5380b4460 0x7fe5380ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53d817640 1 --2- 172.21.15.170:0/305469561 >> v2:172.21.15.177:3300/0 conn(0x7fe53813eb10 0x7fe53813ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53d817640 10 monclient(hunting): get_auth_request con 0x7fe53813eb10 auth_method 0 2023-10-30T17:55:06.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53d817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53d817640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53d817640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53e018640 10 monclient(hunting): get_auth_request con 0x7fe5380b4460 auth_method 0 2023-10-30T17:55:06.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53e018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53e018640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53e018640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53d817640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53d817640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.055+0000 7fe53d817640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.056+0000 7fe53e018640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.056+0000 7fe53e018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.056+0000 7fe53e018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d817640 10 monclient(hunting): handle_auth_done global_id 21466 payload 274 2023-10-30T17:55:06.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d016640 1 -- 172.21.15.170:0/305469561 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941963839 0 0) 0x7fe5380aeda0 con 0x7fe5380b08c0 2023-10-30T17:55:06.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d817640 1 -- 172.21.15.170:0/305469561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5380b4460 msgr2=0x7fe5380ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:06.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d817640 1 --2- 172.21.15.170:0/305469561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5380b4460 0x7fe5380ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d817640 1 -- 172.21.15.170:0/305469561 >> v1:172.21.15.170:6789/0 conn(0x7fe5380b08c0 legacy=0x7fe5380b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d817640 10 monclient: _finish_hunting 0 2023-10-30T17:55:06.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d817640 1 monclient: found mon.noname-c 2023-10-30T17:55:06.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d817640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:06.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d817640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:55:06.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d817640 1 -- 172.21.15.170:0/305469561 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5380af010 con 0x7fe53813eb10 2023-10-30T17:55:06.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.057+0000 7fe53d016640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T17:55:06.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.059+0000 7fe53d817640 1 --2- 172.21.15.170:0/305469561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe53813eb10 0x7fe53813ef50 secure :-1 s=READY pgs=5275 cs=0 l=1 rev1=1 crypto rx=0x7fe5280097d0 tx=0x7fe52802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88d3f0f581939f5c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 1 -- 172.21.15.170:0/305469561 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528031770 con 0x7fe53813eb10 2023-10-30T17:55:06.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:06.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 10 monclient: dump: 2023-10-30T17:55:06.074 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.074 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.074 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.074 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.075 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.075 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.075 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.075 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.075 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.076 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 10 monclient: _finish_auth 0 2023-10-30T17:55:06.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 10 monclient: _check_auth_tickets 2023-10-30T17:55:06.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:06.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 1 -- 172.21.15.170:0/305469561 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5280318d0 con 0x7fe53813eb10 2023-10-30T17:55:06.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:06.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 1 -- 172.21.15.170:0/305469561 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528031c60 con 0x7fe53813eb10 2023-10-30T17:55:06.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 5 monclient: authenticate success, global_id 21466 2023-10-30T17:55:06.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:06.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 4 set_mon_vals no callback set 2023-10-30T17:55:06.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:06.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe53d016640 10 monclient: dump: 2023-10-30T17:55:06.078 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.078 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.079 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.079 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.079 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.079 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.079 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.079 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.080 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.080 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 10 monclient: shutdown 2023-10-30T17:55:06.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:06.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 1 -- 172.21.15.170:0/305469561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe53813eb10 msgr2=0x7fe53813ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 1 --2- 172.21.15.170:0/305469561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe53813eb10 0x7fe53813ef50 secure :-1 s=READY pgs=5275 cs=0 l=1 rev1=1 crypto rx=0x7fe5280097d0 tx=0x7fe52802b810 comp rx=0 tx=0).stop 2023-10-30T17:55:06.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 1 -- 172.21.15.170:0/305469561 shutdown_connections 2023-10-30T17:55:06.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 1 --2- 172.21.15.170:0/305469561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe53813eb10 0x7fe53813ef50 unknown :-1 s=CLOSED pgs=5275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 1 --2- 172.21.15.170:0/305469561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5380b4460 0x7fe5380ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 1 -- 172.21.15.170:0/305469561 >> 172.21.15.170:0/305469561 conn(0x7fe538071b90 msgr2=0x7fe538071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:06.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 1 -- 172.21.15.170:0/305469561 shutdown_connections 2023-10-30T17:55:06.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.063+0000 7fe54027c640 1 -- 172.21.15.170:0/305469561 wait complete. 2023-10-30T17:55:06.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.064+0000 7fe54027c640 10 monclient: build_initial_monmap 2023-10-30T17:55:06.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.064+0000 7fe54027c640 10 monclient: monmap: 2023-10-30T17:55:06.082 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:06.083 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:06.083 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:06.083 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:06.083 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:06.083 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.083 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:06.084 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:06.084 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:06.084 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.064+0000 7fe54027c640 1 librados: starting msgr at 2023-10-30T17:55:06.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.064+0000 7fe54027c640 1 librados: starting objecter 2023-10-30T17:55:06.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.064+0000 7fe54027c640 1 Processor -- start 2023-10-30T17:55:06.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 1 -- start start 2023-10-30T17:55:06.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 1 librados: setting wanted keys 2023-10-30T17:55:06.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 1 librados: calling monclient init 2023-10-30T17:55:06.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient: init 2023-10-30T17:55:06.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:06.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:55:06.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5380b08c0 0x7fe538137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient(hunting): picked mon.noname-c con 0x7fe5380b08c0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:06.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5380b4460 0x7fe538139120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient(hunting): picked mon.noname-b con 0x7fe5380b4460 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:06.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe53813eb10 0x7fe538139660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient(hunting): picked mon.noname-a con 0x7fe53813eb10 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:06.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe538138140 con 0x7fe53813eb10 2023-10-30T17:55:06.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe538139d00 con 0x7fe5380b4460 2023-10-30T17:55:06.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe538141dc0 con 0x7fe5380b08c0 2023-10-30T17:55:06.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:06.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.065+0000 7fe54027c640 10 monclient(hunting): authenticate will time out at 2688.388694s 2023-10-30T17:55:06.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e819640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe53813eb10 0x7fe538139660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e018640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5380b08c0 0x7fe538137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53d817640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5380b4460 0x7fe538139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e819640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe53813eb10 0x7fe538139660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40460/0 (socket says 172.21.15.170:40460) 2023-10-30T17:55:06.089 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e819640 1 -- 172.21.15.170:0/1034455436 learned_addr learned my addr 172.21.15.170:0/1034455436 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:06.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e819640 10 monclient(hunting): get_auth_request con 0x7fe53813eb10 auth_method 0 2023-10-30T17:55:06.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e819640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e819640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53d817640 10 monclient(hunting): get_auth_request con 0x7fe5380b4460 auth_method 0 2023-10-30T17:55:06.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53d817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53d817640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53d817640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e018640 10 monclient(hunting): get_auth_request con 0x7fe5380b08c0 auth_method 0 2023-10-30T17:55:06.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e018640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e018640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e819640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53e819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53d817640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53d817640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.066+0000 7fe53d817640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.067+0000 7fe53e018640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.067+0000 7fe53e018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.067+0000 7fe53e018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53e819640 10 monclient(hunting): handle_auth_done global_id 16023 payload 824 2023-10-30T17:55:06.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53e819640 1 -- 172.21.15.170:0/1034455436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5380b08c0 msgr2=0x7fe538137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53e819640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5380b08c0 0x7fe538137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53e819640 1 -- 172.21.15.170:0/1034455436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5380b4460 msgr2=0x7fe538139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53e819640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5380b4460 0x7fe538139120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53e819640 10 monclient: _finish_hunting 0 2023-10-30T17:55:06.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53e819640 1 monclient: found mon.noname-a 2023-10-30T17:55:06.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53e819640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:06.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53e819640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:06.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53e819640 1 -- 172.21.15.170:0/1034455436 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528009400 con 0x7fe53813eb10 2023-10-30T17:55:06.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.069+0000 7fe53d817640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5380b4460 0x7fe538139120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T17:55:06.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.072+0000 7fe53e819640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe53813eb10 0x7fe538139660 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7fe53000c7e0 tx=0x7fe53000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.072+0000 7fe52effd640 10 client.?.objecter ms_handle_connect 0x7fe53813eb10 2023-10-30T17:55:06.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.072+0000 7fe52effd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:06.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 1 -- 172.21.15.170:0/1034455436 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53000a640 con 0x7fe53813eb10 2023-10-30T17:55:06.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:06.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 10 monclient: dump: 2023-10-30T17:55:06.099 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.099 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.099 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.099 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.099 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.100 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.100 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.100 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.100 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.100 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 10 monclient: _finish_auth 0 2023-10-30T17:55:06.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 10 monclient: _check_auth_tickets 2023-10-30T17:55:06.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:06.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 1 -- 172.21.15.170:0/1034455436 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53000a7c0 con 0x7fe53813eb10 2023-10-30T17:55:06.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe54027c640 5 monclient: authenticate success, global_id 16023 2023-10-30T17:55:06.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe54027c640 10 monclient: _renew_subs 2023-10-30T17:55:06.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe54027c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:06.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5380b3b30 con 0x7fe53813eb10 2023-10-30T17:55:06.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:06.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 1 -- 172.21.15.170:0/1034455436 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53000ab70 con 0x7fe53813eb10 2023-10-30T17:55:06.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:06.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe52effd640 10 monclient: dump: 2023-10-30T17:55:06.103 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.103 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.103 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.104 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.104 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.104 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.104 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.104 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.105 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.105 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe53c815640 4 set_mon_vals no callback set 2023-10-30T17:55:06.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe54027c640 10 client.16023.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:06.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe54027c640 10 monclient: _renew_subs 2023-10-30T17:55:06.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe54027c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:06.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe538142340 con 0x7fe53813eb10 2023-10-30T17:55:06.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.075+0000 7fe54027c640 1 librados: init done 2023-10-30T17:55:06.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.076+0000 7fe54027c640 10 client.16023.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:06.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.076+0000 7fe54027c640 15 client.16023.objecter _session_command_op_assign -1 1 2023-10-30T17:55:06.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.076+0000 7fe54027c640 20 client.16023.objecter _get_session osd=-1 returning homeless 2023-10-30T17:55:06.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.076+0000 7fe54027c640 10 client.16023.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:06.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.076+0000 7fe54027c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:06.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.076+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe53813f490 con 0x7fe53813eb10 2023-10-30T17:55:06.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.095+0000 7fe52effd640 1 -- 172.21.15.170:0/1034455436 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe53000acd0 con 0x7fe53813eb10 2023-10-30T17:55:06.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.096+0000 7fe52effd640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fe51c06c850 0x7fe51c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.096+0000 7fe53e018640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fe51c06c850 0x7fe51c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.096+0000 7fe53e018640 10 monclient: get_auth_request con 0x7fe51c06c850 auth_method 0 2023-10-30T17:55:06.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.096+0000 7fe53e018640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fe51c06c850 0x7fe51c06ed10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe534009b40 tx=0x7fe534009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.096+0000 7fe52effd640 10 client.16023.objecter ms_handle_connect 0x7fe51c06c850 2023-10-30T17:55:06.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 1 -- 172.21.15.170:0/1034455436 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fe530058400 con 0x7fe53813eb10 2023-10-30T17:55:06.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 10 client.16023.objecter ms_dispatch 0x7fe538000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:06.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 3 client.16023.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:06.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 3 client.16023.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:06.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 10 client.16023.objecter checking command 1 2023-10-30T17:55:06.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] conn(0x7fe51c074060 0x7fe51c076500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 20 client.16023.objecter _get_session s=0x7fe51c073990 osd=2 3 2023-10-30T17:55:06.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 20 client.16023.objecter put_session s=0x7fe51c073990 osd=2 3 2023-10-30T17:55:06.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 15 client.16023.objecter _session_command_op_remove -1 1 2023-10-30T17:55:06.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 20 client.16023.objecter _get_session s=0x7fe51c073990 osd=2 3 2023-10-30T17:55:06.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 20 client.16023.objecter get_session s=0x7fe51c073990 osd=2 3 2023-10-30T17:55:06.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 15 client.16023.objecter _session_command_op_assign 2 1 2023-10-30T17:55:06.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 20 client.16023.objecter put_session s=0x7fe51c073990 osd=2 4 2023-10-30T17:55:06.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 10 client.16023.objecter _send_command 1 2023-10-30T17:55:06.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 1 -- 172.21.15.170:0/1034455436 --> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe51c076bd0 con 0x7fe51c074060 2023-10-30T17:55:06.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 20 client.16023.objecter dump_active .. 0 homeless 2023-10-30T17:55:06.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 1 -- 172.21.15.170:0/1034455436 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=914) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe530001890 con 0x7fe53813eb10 2023-10-30T17:55:06.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.098+0000 7fe52effd640 10 monclient: handle_get_version_reply finishing 1 version 914 2023-10-30T17:55:06.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.099+0000 7fe53d817640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] conn(0x7fe51c074060 0x7fe51c076500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.099+0000 7fe53d817640 10 monclient: get_auth_request con 0x7fe51c074060 auth_method 0 2023-10-30T17:55:06.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.102+0000 7fe53d817640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] conn(0x7fe51c074060 0x7fe51c076500 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.102+0000 7fe52effd640 10 client.16023.objecter ms_handle_connect 0x7fe51c074060 2023-10-30T17:55:06.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.124+0000 7fe52effd640 1 -- 172.21.15.170:0/1034455436 <== osd.2 v2:172.21.15.170:6816/425342180 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fe51c076bd0 con 0x7fe51c074060 2023-10-30T17:55:06.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.124+0000 7fe52effd640 10 client.16023.objecter ms_dispatch 0x7fe538000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:55:06.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.124+0000 7fe52effd640 10 client.16023.objecter _finish_command 1 = system:0 2023-10-30T17:55:06.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.124+0000 7fe52effd640 20 client.16023.objecter put_session s=0x7fe51c073990 osd=2 4 2023-10-30T17:55:06.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.124+0000 7fe52effd640 15 client.16023.objecter _session_command_op_remove 2 1 2023-10-30T17:55:06.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 10 client.16023.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T17:55:06.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 15 client.16023.objecter _session_command_op_assign -1 2 2023-10-30T17:55:06.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 20 client.16023.objecter _get_session s=0x7fe51c073990 osd=2 3 2023-10-30T17:55:06.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 20 client.16023.objecter put_session s=0x7fe51c073990 osd=2 3 2023-10-30T17:55:06.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 20 client.16023.objecter _get_session s=0x7fe51c073990 osd=2 3 2023-10-30T17:55:06.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 15 client.16023.objecter _session_command_op_remove -1 2 2023-10-30T17:55:06.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 20 client.16023.objecter get_session s=0x7fe51c073990 osd=2 3 2023-10-30T17:55:06.139 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 15 client.16023.objecter _session_command_op_assign 2 2 2023-10-30T17:55:06.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 20 client.16023.objecter put_session s=0x7fe51c073990 osd=2 4 2023-10-30T17:55:06.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 10 client.16023.objecter _send_command 2 2023-10-30T17:55:06.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.137+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 --> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe538067cd0 con 0x7fe51c074060 2023-10-30T17:55:06.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.151+0000 7fe52effd640 1 -- 172.21.15.170:0/1034455436 <== osd.2 v2:172.21.15.170:6816/425342180 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe538067cd0 con 0x7fe51c074060 2023-10-30T17:55:06.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.151+0000 7fe52effd640 10 client.16023.objecter ms_dispatch 0x7fe538000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:55:06.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.151+0000 7fe52effd640 10 client.16023.objecter _finish_command 2 = system:0 2023-10-30T17:55:06.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.151+0000 7fe52effd640 20 client.16023.objecter put_session s=0x7fe51c073990 osd=2 4 2023-10-30T17:55:06.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.151+0000 7fe52effd640 15 client.16023.objecter _session_command_op_remove 2 2 2023-10-30T17:55:06.154 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-30T17:55:06.154 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T17:55:06.154 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-30T17:55:06.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.151+0000 7fe54027c640 10 librados: watch_flush enter 2023-10-30T17:55:06.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 10 librados: watch_flush exit 2023-10-30T17:55:06.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 10 client.16023.objecter close_session for osd.2 2023-10-30T17:55:06.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 >> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] conn(0x7fe51c074060 msgr2=0x7fe51c076500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.170:6816/425342180,v1:172.21.15.170:6817/425342180] conn(0x7fe51c074060 0x7fe51c076500 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 20 client.16023.objecter put_session s=0x7fe51c073990 osd=2 1 2023-10-30T17:55:06.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 20 client.16023.objecter shutdown clearing up homeless session... 2023-10-30T17:55:06.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 10 client.16023.objecter successfully canceled tick 2023-10-30T17:55:06.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fe51c06c850 msgr2=0x7fe51c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fe51c06c850 0x7fe51c06ed10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe534009b40 tx=0x7fe534009190 comp rx=0 tx=0).stop 2023-10-30T17:55:06.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 10 monclient: shutdown 2023-10-30T17:55:06.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:06.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe53813eb10 msgr2=0x7fe538139660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 --2- 172.21.15.170:0/1034455436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe53813eb10 0x7fe538139660 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7fe53000c7e0 tx=0x7fe53000cca0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe53e819640 1 -- 172.21.15.170:0/1034455436 reap_dead start 2023-10-30T17:55:06.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 shutdown_connections 2023-10-30T17:55:06.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 >> 172.21.15.170:0/1034455436 conn(0x7fe538071b90 msgr2=0x7fe5381349d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:06.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 shutdown_connections 2023-10-30T17:55:06.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 -- 172.21.15.170:0/1034455436 wait complete. 2023-10-30T17:55:06.162 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.152+0000 7fe54027c640 1 librados: shutdown 2023-10-30T17:55:06.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T17:55:06.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.267+0000 7ff30c44b640 10 monclient: get_monmap_and_config 2023-10-30T17:55:06.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.267+0000 7ff30c44b640 10 monclient: build_initial_monmap 2023-10-30T17:55:06.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.267+0000 7ff30c44b640 10 monclient: monmap: 2023-10-30T17:55:06.269 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:06.270 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.270 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:06.268705+0000 2023-10-30T17:55:06.270 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:06.268705+0000 2023-10-30T17:55:06.270 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:06.270 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.271 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:06.271 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:06.271 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:06.271 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:06.271 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:06.272 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.268+0000 7ff30c44b640 1 Processor -- start 2023-10-30T17:55:06.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.268+0000 7ff30c44b640 1 -- start start 2023-10-30T17:55:06.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.268+0000 7ff30c44b640 10 monclient: init 2023-10-30T17:55:06.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:06.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-10-30T17:55:06.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b4440 0x7ff3040ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 10 monclient(hunting): picked mon.noname-b con 0x7ff3040b4440 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:06.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3040b0930 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:06.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff30413e6c0 0x7ff30413eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 10 monclient(hunting): picked mon.noname-a con 0x7ff30413e6c0 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:06.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3040b1480 con 0x7ff3040b0930 2023-10-30T17:55:06.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3040b4820 con 0x7ff30413e6c0 2023-10-30T17:55:06.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040af360 con 0x7ff3040b4440 2023-10-30T17:55:06.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:06.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30c44b640 10 monclient(hunting): authenticate will time out at 2688.592395s 2023-10-30T17:55:06.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.269+0000 7ff30a1e7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b4440 0x7ff3040ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.270+0000 7ff30a1e7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b4440 0x7ff3040ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:38996/0 (socket says 172.21.15.170:38996) 2023-10-30T17:55:06.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.270+0000 7ff30a1e7640 1 -- 172.21.15.170:0/754989491 learned_addr learned my addr 172.21.15.170:0/754989491 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:06.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.270+0000 7ff30a1e7640 10 monclient(hunting): get_auth_request con 0x7ff3040b4440 auth_method 0 2023-10-30T17:55:06.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.270+0000 7ff30a1e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.270+0000 7ff30a1e7640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.270+0000 7ff30a1e7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.270+0000 7ff30a1e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.270+0000 7ff30a1e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.270+0000 7ff30a1e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.272+0000 7ff30a1e7640 10 monclient(hunting): handle_auth_done global_id 16334 payload 274 2023-10-30T17:55:06.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.272+0000 7ff30a1e7640 1 -- 172.21.15.170:0/754989491 >> v2:172.21.15.170:3300/0 conn(0x7ff30413e6c0 msgr2=0x7ff30413eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:06.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.272+0000 7ff30a1e7640 1 --2- 172.21.15.170:0/754989491 >> v2:172.21.15.170:3300/0 conn(0x7ff30413e6c0 0x7ff30413eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.272+0000 7ff30a1e7640 1 -- 172.21.15.170:0/754989491 >> v1:172.21.15.170:6789/0 conn(0x7ff3040b0930 legacy=0x7ff3040b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:06.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.272+0000 7ff30a1e7640 10 monclient: _finish_hunting 0 2023-10-30T17:55:06.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.272+0000 7ff30a1e7640 1 monclient: found mon.noname-b 2023-10-30T17:55:06.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.272+0000 7ff30a1e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:06.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.272+0000 7ff30a1e7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:55:06.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.272+0000 7ff30a1e7640 1 -- 172.21.15.170:0/754989491 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3040b0510 con 0x7ff3040b4440 2023-10-30T17:55:06.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.274+0000 7ff30a1e7640 1 --2- 172.21.15.170:0/754989491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b4440 0x7ff3040ae7e0 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7ff3040b3380 tx=0x7ff2f402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=884c40936b187c7d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 1 -- 172.21.15.170:0/754989491 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4031640 con 0x7ff3040b4440 2023-10-30T17:55:06.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:06.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 10 monclient: dump: 2023-10-30T17:55:06.287 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.287 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.287 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.287 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.287 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.288 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.288 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.288 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.288 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.289 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 10 monclient: _finish_auth 0 2023-10-30T17:55:06.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 10 monclient: _check_auth_tickets 2023-10-30T17:55:06.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:06.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 1 -- 172.21.15.170:0/754989491 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f40317a0 con 0x7ff3040b4440 2023-10-30T17:55:06.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:06.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 1 -- 172.21.15.170:0/754989491 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4031b30 con 0x7ff3040b4440 2023-10-30T17:55:06.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff30c44b640 5 monclient: authenticate success, global_id 16334 2023-10-30T17:55:06.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff30c44b640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:06.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff30c44b640 4 set_mon_vals no callback set 2023-10-30T17:55:06.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:55:06.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.280+0000 7ff3091e5640 10 monclient: dump: 2023-10-30T17:55:06.291 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.291 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.292 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.292 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.292 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.292 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.293 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.293 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.293 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.293 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.281+0000 7ff30c44b640 10 monclient: shutdown 2023-10-30T17:55:06.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.281+0000 7ff30c44b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:06.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.281+0000 7ff30c44b640 1 -- 172.21.15.170:0/754989491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b4440 msgr2=0x7ff3040ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.281+0000 7ff30c44b640 1 --2- 172.21.15.170:0/754989491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b4440 0x7ff3040ae7e0 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7ff3040b3380 tx=0x7ff2f402b8f0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.281+0000 7ff30c44b640 1 -- 172.21.15.170:0/754989491 shutdown_connections 2023-10-30T17:55:06.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.281+0000 7ff30c44b640 1 --2- 172.21.15.170:0/754989491 >> v2:172.21.15.170:3300/0 conn(0x7ff30413e6c0 0x7ff30413eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.281+0000 7ff30c44b640 1 --2- 172.21.15.170:0/754989491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b4440 0x7ff3040ae7e0 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.281+0000 7ff30c44b640 1 -- 172.21.15.170:0/754989491 >> 172.21.15.170:0/754989491 conn(0x7ff304071b60 msgr2=0x7ff304071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:06.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.281+0000 7ff30c44b640 1 -- 172.21.15.170:0/754989491 shutdown_connections 2023-10-30T17:55:06.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.281+0000 7ff30c44b640 1 -- 172.21.15.170:0/754989491 wait complete. 2023-10-30T17:55:06.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.282+0000 7ff30c44b640 10 monclient: build_initial_monmap 2023-10-30T17:55:06.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.282+0000 7ff30c44b640 10 monclient: monmap: 2023-10-30T17:55:06.296 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:06.296 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:06.296 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:06.296 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:06.297 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:06.297 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.297 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:06.297 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:06.297 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:06.297 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.282+0000 7ff30c44b640 1 librados: starting msgr at 2023-10-30T17:55:06.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.282+0000 7ff30c44b640 1 librados: starting objecter 2023-10-30T17:55:06.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.282+0000 7ff30c44b640 1 Processor -- start 2023-10-30T17:55:06.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.282+0000 7ff30c44b640 1 -- start start 2023-10-30T17:55:06.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.282+0000 7ff30c44b640 1 librados: setting wanted keys 2023-10-30T17:55:06.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.282+0000 7ff30c44b640 1 librados: calling monclient init 2023-10-30T17:55:06.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.282+0000 7ff30c44b640 10 monclient: init 2023-10-30T17:55:06.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:06.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:55:06.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b0930 0x7ff304140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 10 monclient(hunting): picked mon.noname-c con 0x7ff3040b0930 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:06.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3040b4440 0x7ff3040abac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 10 monclient(hunting): picked mon.noname-b con 0x7ff3040b4440 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:06.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff30413e6c0 0x7ff3040ac000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 10 monclient(hunting): picked mon.noname-a con 0x7ff30413e6c0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:06.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff304142ad0 con 0x7ff30413e6c0 2023-10-30T17:55:06.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff304141cf0 con 0x7ff3040b4440 2023-10-30T17:55:06.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff304142d40 con 0x7ff3040b0930 2023-10-30T17:55:06.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:06.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30c44b640 10 monclient(hunting): authenticate will time out at 2688.606237s 2023-10-30T17:55:06.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30a9e8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff30413e6c0 0x7ff3040ac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30a9e8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff30413e6c0 0x7ff3040ac000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40470/0 (socket says 172.21.15.170:40470) 2023-10-30T17:55:06.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30a9e8640 1 -- 172.21.15.170:0/192647540 learned_addr learned my addr 172.21.15.170:0/192647540 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:06.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30a9e8640 10 monclient(hunting): get_auth_request con 0x7ff30413e6c0 auth_method 0 2023-10-30T17:55:06.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30a9e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30a9e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.283+0000 7ff30a9e8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.284+0000 7ff30a9e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.284+0000 7ff30a9e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.284+0000 7ff30a9e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.306 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.287+0000 7ff30a9e8640 10 monclient(hunting): handle_auth_done global_id 16026 payload 824 2023-10-30T17:55:06.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.287+0000 7ff30a9e8640 1 -- 172.21.15.170:0/192647540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b0930 msgr2=0x7ff304140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.287+0000 7ff30a9e8640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b0930 0x7ff304140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.287+0000 7ff30a9e8640 1 -- 172.21.15.170:0/192647540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3040b4440 msgr2=0x7ff3040abac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.287+0000 7ff30a9e8640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3040b4440 0x7ff3040abac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.287+0000 7ff30a9e8640 10 monclient: _finish_hunting 0 2023-10-30T17:55:06.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.287+0000 7ff30a9e8640 1 monclient: found mon.noname-a 2023-10-30T17:55:06.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.287+0000 7ff30a9e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:06.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.287+0000 7ff30a9e8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:06.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.287+0000 7ff30a9e8640 1 -- 172.21.15.170:0/192647540 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f40094d0 con 0x7ff30413e6c0 2023-10-30T17:55:06.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.289+0000 7ff30a9e8640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff30413e6c0 0x7ff3040ac000 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00b970 tx=0x7ff2fc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.289+0000 7ff2faffd640 10 client.?.objecter ms_handle_connect 0x7ff30413e6c0 2023-10-30T17:55:06.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.289+0000 7ff2faffd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:06.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff2faffd640 1 -- 172.21.15.170:0/192647540 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc00cb40 con 0x7ff30413e6c0 2023-10-30T17:55:06.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff2faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.309 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff2faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:06.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff2faffd640 10 monclient: dump: 2023-10-30T17:55:06.310 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.310 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.310 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.310 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.311 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.311 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.311 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.311 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.311 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.312 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff2faffd640 10 monclient: _finish_auth 0 2023-10-30T17:55:06.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff2faffd640 10 monclient: _check_auth_tickets 2023-10-30T17:55:06.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff2faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:06.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff2faffd640 1 -- 172.21.15.170:0/192647540 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2fc00ccc0 con 0x7ff30413e6c0 2023-10-30T17:55:06.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff30c44b640 5 monclient: authenticate success, global_id 16026 2023-10-30T17:55:06.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff30c44b640 10 monclient: _renew_subs 2023-10-30T17:55:06.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff30c44b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:06.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.292+0000 7ff30c44b640 1 -- 172.21.15.170:0/192647540 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30413faa0 con 0x7ff30413e6c0 2023-10-30T17:55:06.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff2faffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:06.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff2faffd640 1 -- 172.21.15.170:0/192647540 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc014680 con 0x7ff30413e6c0 2023-10-30T17:55:06.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff2faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff2faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:06.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff2faffd640 10 monclient: dump: 2023-10-30T17:55:06.314 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.315 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.315 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.315 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.315 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.315 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.316 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.316 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.316 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.316 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff3089e4640 4 set_mon_vals no callback set 2023-10-30T17:55:06.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff30c44b640 10 client.16026.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:06.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff30c44b640 10 monclient: _renew_subs 2023-10-30T17:55:06.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff30c44b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:06.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff30c44b640 1 -- 172.21.15.170:0/192647540 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff304141220 con 0x7ff30413e6c0 2023-10-30T17:55:06.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff30c44b640 1 librados: init done 2023-10-30T17:55:06.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.293+0000 7ff30c44b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:06.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.294+0000 7ff30c44b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:06.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.294+0000 7ff30c44b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:06.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.294+0000 7ff30c44b640 1 -- 172.21.15.170:0/192647540 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff304141920 con 0x7ff30413e6c0 2023-10-30T17:55:06.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.533+0000 7ff2faffd640 1 -- 172.21.15.170:0/192647540 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff2fc01d3b0 con 0x7ff30413e6c0 2023-10-30T17:55:06.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.533+0000 7ff2faffd640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff2e806c850 0x7ff2e806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.534+0000 7ff30a1e7640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff2e806c850 0x7ff2e806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.534+0000 7ff30a1e7640 10 monclient: get_auth_request con 0x7ff2e806c850 auth_method 0 2023-10-30T17:55:06.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.534+0000 7ff30a1e7640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff2e806c850 0x7ff2e806ed10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4036000 tx=0x7ff2f4030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.534+0000 7ff2faffd640 10 client.16026.objecter ms_handle_connect 0x7ff2e806c850 2023-10-30T17:55:06.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.547+0000 7ff2faffd640 1 -- 172.21.15.170:0/192647540 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7ff2fc00ce30 con 0x7ff30413e6c0 2023-10-30T17:55:06.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.547+0000 7ff2faffd640 10 client.16026.objecter ms_dispatch 0x7ff304000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:06.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.547+0000 7ff2faffd640 3 client.16026.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:06.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.547+0000 7ff2faffd640 3 client.16026.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:06.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.547+0000 7ff2faffd640 20 client.16026.objecter dump_active .. 0 homeless 2023-10-30T17:55:06.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.547+0000 7ff2faffd640 1 -- 172.21.15.170:0/192647540 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff2fc054ac0 con 0x7ff30413e6c0 2023-10-30T17:55:06.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.547+0000 7ff2faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:06.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.547+0000 7ff2faffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:06.729 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.728+0000 7ff30c44b640 1 -- 172.21.15.170:0/192647540 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff30413f070 con 0x7ff2e806c850 2023-10-30T17:55:06.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.728+0000 7ff2faffd640 1 -- 172.21.15.170:0/192647540 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff2f4031990 con 0x7ff2e806c850 2023-10-30T17:55:06.730 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T17:55:06.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.730+0000 7ff30c44b640 10 librados: watch_flush enter 2023-10-30T17:55:06.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.730+0000 7ff30c44b640 10 librados: watch_flush exit 2023-10-30T17:55:06.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.730+0000 7ff30c44b640 20 client.16026.objecter shutdown clearing up homeless session... 2023-10-30T17:55:06.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.730+0000 7ff30c44b640 10 client.16026.objecter successfully canceled tick 2023-10-30T17:55:06.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 -- 172.21.15.170:0/192647540 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff2e806c850 msgr2=0x7ff2e806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff2e806c850 0x7ff2e806ed10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4036000 tx=0x7ff2f4030000 comp rx=0 tx=0).stop 2023-10-30T17:55:06.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 10 monclient: shutdown 2023-10-30T17:55:06.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:06.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 -- 172.21.15.170:0/192647540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff30413e6c0 msgr2=0x7ff3040ac000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff30413e6c0 0x7ff3040ac000 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00b970 tx=0x7ff2fc00be30 comp rx=0 tx=0).stop 2023-10-30T17:55:06.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 -- 172.21.15.170:0/192647540 shutdown_connections 2023-10-30T17:55:06.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff2e806c850 0x7ff2e806ed10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3040b4440 0x7ff3040abac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff30413e6c0 0x7ff3040ac000 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 --2- 172.21.15.170:0/192647540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3040b0930 0x7ff304140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 -- 172.21.15.170:0/192647540 >> 172.21.15.170:0/192647540 conn(0x7ff304071b60 msgr2=0x7ff30413c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:06.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 -- 172.21.15.170:0/192647540 shutdown_connections 2023-10-30T17:55:06.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 -- 172.21.15.170:0/192647540 wait complete. 2023-10-30T17:55:06.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.731+0000 7ff30c44b640 1 librados: shutdown 2023-10-30T17:55:06.743 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T17:55:06.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T17:55:06.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.845+0000 7efd3f27c640 10 monclient: get_monmap_and_config 2023-10-30T17:55:06.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.845+0000 7efd3f27c640 10 monclient: build_initial_monmap 2023-10-30T17:55:06.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.845+0000 7efd3f27c640 10 monclient: monmap: 2023-10-30T17:55:06.847 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:06.847 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.848 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:06.846564+0000 2023-10-30T17:55:06.848 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:06.846564+0000 2023-10-30T17:55:06.848 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:06.848 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.848 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:06.849 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:06.849 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:06.849 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:06.849 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:06.849 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 1 Processor -- start 2023-10-30T17:55:06.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 1 -- start start 2023-10-30T17:55:06.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 10 monclient: init 2023-10-30T17:55:06.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:06.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-10-30T17:55:06.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd3806bc80 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:06.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd380abd30 0x7efd3806ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 10 monclient(hunting): picked mon.noname-a con 0x7efd380abd30 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:06.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd380707e0 0x7efd38070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 10 monclient(hunting): picked mon.noname-b con 0x7efd380707e0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:06.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3806c060 con 0x7efd3806bc80 2023-10-30T17:55:06.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3806fe00 con 0x7efd380abd30 2023-10-30T17:55:06.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd380700e0 con 0x7efd380707e0 2023-10-30T17:55:06.855 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:06.855 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.846+0000 7efd3f27c640 10 monclient(hunting): authenticate will time out at 2689.170030s 2023-10-30T17:55:06.855 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3c817640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd380abd30 0x7efd3806ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.855 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3d018640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7efd3806bc80 0x7efd380afa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:45930/0 (socket says 172.21.15.170:45930) 2023-10-30T17:55:06.855 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3d018640 1 -- 172.21.15.170:0/2690337204 learned_addr learned my addr 172.21.15.170:0/2690337204 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:06.856 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3d819640 1 --2- 172.21.15.170:0/2690337204 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd380707e0 0x7efd38070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.856 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3c817640 10 monclient(hunting): get_auth_request con 0x7efd380abd30 auth_method 0 2023-10-30T17:55:06.856 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3c817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.856 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3c817640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.856 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3c817640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.857 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3d819640 10 monclient(hunting): get_auth_request con 0x7efd380707e0 auth_method 0 2023-10-30T17:55:06.857 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3d819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.857 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3d819640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.857 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.847+0000 7efd3d819640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.857 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.848+0000 7efd3c817640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.848+0000 7efd3c817640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.848+0000 7efd3c817640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.848+0000 7efd3d819640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.848+0000 7efd3d819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.848+0000 7efd3d819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.849+0000 7efd3c817640 10 monclient(hunting): handle_auth_done global_id 16029 payload 274 2023-10-30T17:55:06.859 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.849+0000 7efd3c817640 1 -- 172.21.15.170:0/2690337204 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd380707e0 msgr2=0x7efd38070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:06.859 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.849+0000 7efd3c817640 1 --2- 172.21.15.170:0/2690337204 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd380707e0 0x7efd38070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.859 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.849+0000 7efd3c817640 1 -- 172.21.15.170:0/2690337204 >> v1:172.21.15.177:6789/0 conn(0x7efd3806bc80 legacy=0x7efd380afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.859 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.849+0000 7efd2ffff640 1 -- 172.21.15.170:0/2690337204 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260159673 0 0) 0x7efd3806c060 con 0x7efd3806bc80 2023-10-30T17:55:06.859 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.850+0000 7efd3c817640 10 monclient: _finish_hunting 0 2023-10-30T17:55:06.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.850+0000 7efd3c817640 1 monclient: found mon.noname-a 2023-10-30T17:55:06.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.850+0000 7efd3c817640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:06.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.850+0000 7efd3c817640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:06.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.850+0000 7efd3c817640 1 -- 172.21.15.170:0/2690337204 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd380703c0 con 0x7efd380abd30 2023-10-30T17:55:06.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.850+0000 7efd2ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T17:55:06.864 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.852+0000 7efd3c817640 1 --2- 172.21.15.170:0/2690337204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd380abd30 0x7efd3806ca10 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7efd280097a0 tx=0x7efd2802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e7d89020d6e77f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.864 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 1 -- 172.21.15.170:0/2690337204 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2802e070 con 0x7efd380abd30 2023-10-30T17:55:06.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:06.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 10 monclient: dump: 2023-10-30T17:55:06.865 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.865 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.866 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.866 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.866 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.866 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.866 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.867 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.867 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.867 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 10 monclient: _finish_auth 0 2023-10-30T17:55:06.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 10 monclient: _check_auth_tickets 2023-10-30T17:55:06.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:06.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 1 -- 172.21.15.170:0/2690337204 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd280315a0 con 0x7efd380abd30 2023-10-30T17:55:06.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd3f27c640 5 monclient: authenticate success, global_id 16029 2023-10-30T17:55:06.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd3f27c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:06.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:06.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd3f27c640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:06.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 1 -- 172.21.15.170:0/2690337204 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd28031930 con 0x7efd380abd30 2023-10-30T17:55:06.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd3f27c640 4 set_mon_vals no callback set 2023-10-30T17:55:06.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:06.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.855+0000 7efd2ffff640 10 monclient: dump: 2023-10-30T17:55:06.870 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.870 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.870 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.870 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.870 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.871 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.871 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.871 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.871 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.871 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.856+0000 7efd3f27c640 10 monclient: shutdown 2023-10-30T17:55:06.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.856+0000 7efd3f27c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:06.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.856+0000 7efd3f27c640 1 -- 172.21.15.170:0/2690337204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd380abd30 msgr2=0x7efd3806ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.856+0000 7efd3f27c640 1 --2- 172.21.15.170:0/2690337204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd380abd30 0x7efd3806ca10 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7efd280097a0 tx=0x7efd2802b840 comp rx=0 tx=0).stop 2023-10-30T17:55:06.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.856+0000 7efd3f27c640 1 -- 172.21.15.170:0/2690337204 shutdown_connections 2023-10-30T17:55:06.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.856+0000 7efd3f27c640 1 --2- 172.21.15.170:0/2690337204 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd380707e0 0x7efd38070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.856+0000 7efd3f27c640 1 --2- 172.21.15.170:0/2690337204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd380abd30 0x7efd3806ca10 unknown :-1 s=CLOSED pgs=3374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.856+0000 7efd3f27c640 1 -- 172.21.15.170:0/2690337204 >> 172.21.15.170:0/2690337204 conn(0x7efd380b5270 msgr2=0x7efd380b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:06.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.856+0000 7efd3f27c640 1 -- 172.21.15.170:0/2690337204 shutdown_connections 2023-10-30T17:55:06.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.856+0000 7efd3f27c640 1 -- 172.21.15.170:0/2690337204 wait complete. 2023-10-30T17:55:06.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.857+0000 7efd3f27c640 10 monclient: build_initial_monmap 2023-10-30T17:55:06.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.857+0000 7efd3f27c640 10 monclient: monmap: 2023-10-30T17:55:06.874 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:06.874 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:06.874 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:06.874 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:06.875 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:06.875 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.875 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:06.875 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:06.875 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:06.876 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.857+0000 7efd3f27c640 1 librados: starting msgr at 2023-10-30T17:55:06.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.857+0000 7efd3f27c640 1 librados: starting objecter 2023-10-30T17:55:06.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.857+0000 7efd3f27c640 1 Processor -- start 2023-10-30T17:55:06.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.857+0000 7efd3f27c640 1 -- start start 2023-10-30T17:55:06.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.857+0000 7efd3f27c640 1 librados: setting wanted keys 2023-10-30T17:55:06.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.857+0000 7efd3f27c640 1 librados: calling monclient init 2023-10-30T17:55:06.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.857+0000 7efd3f27c640 10 monclient: init 2023-10-30T17:55:06.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:06.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:55:06.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd3806bc80 0x7efd380adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 10 monclient(hunting): picked mon.noname-b con 0x7efd3806bc80 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:06.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd380707e0 0x7efd381d7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 10 monclient(hunting): picked mon.noname-a con 0x7efd380707e0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:06.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd380abd30 0x7efd381d7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 10 monclient(hunting): picked mon.noname-c con 0x7efd380abd30 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:06.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd381d71e0 con 0x7efd380707e0 2023-10-30T17:55:06.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd380aee00 con 0x7efd3806bc80 2023-10-30T17:55:06.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:06.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd38142950 con 0x7efd380abd30 2023-10-30T17:55:06.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:06.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3f27c640 10 monclient(hunting): authenticate will time out at 2689.181239s 2023-10-30T17:55:06.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3d819640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd380abd30 0x7efd381d7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3c817640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd380707e0 0x7efd381d7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3d018640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd3806bc80 0x7efd380adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3c817640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd380707e0 0x7efd381d7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40484/0 (socket says 172.21.15.170:40484) 2023-10-30T17:55:06.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3c817640 1 -- 172.21.15.170:0/2712766467 learned_addr learned my addr 172.21.15.170:0/2712766467 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:06.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3c817640 10 monclient(hunting): get_auth_request con 0x7efd380707e0 auth_method 0 2023-10-30T17:55:06.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3c817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3c817640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.858+0000 7efd3c817640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d018640 10 monclient(hunting): get_auth_request con 0x7efd3806bc80 auth_method 0 2023-10-30T17:55:06.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d018640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d018640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d819640 10 monclient(hunting): get_auth_request con 0x7efd380abd30 auth_method 0 2023-10-30T17:55:06.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:06.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d819640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:06.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d819640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:06.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3c817640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3c817640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3c817640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d018640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d819640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:06.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:06.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.859+0000 7efd3d819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:06.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.862+0000 7efd3d018640 10 monclient(hunting): handle_auth_done global_id 21475 payload 824 2023-10-30T17:55:06.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.862+0000 7efd3d018640 1 -- 172.21.15.170:0/2712766467 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd380abd30 msgr2=0x7efd381d7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.862+0000 7efd3d018640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd380abd30 0x7efd381d7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.862+0000 7efd3d018640 1 -- 172.21.15.170:0/2712766467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd380707e0 msgr2=0x7efd381d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.862+0000 7efd3d018640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd380707e0 0x7efd381d7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.862+0000 7efd3d018640 10 monclient: _finish_hunting 0 2023-10-30T17:55:06.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.862+0000 7efd3d018640 1 monclient: found mon.noname-b 2023-10-30T17:55:06.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.862+0000 7efd3d018640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:06.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.862+0000 7efd3d018640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:06.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.862+0000 7efd3d018640 1 -- 172.21.15.170:0/2712766467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd28009400 con 0x7efd3806bc80 2023-10-30T17:55:06.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.864+0000 7efd3d018640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd3806bc80 0x7efd380adb00 secure :-1 s=READY pgs=5277 cs=0 l=1 rev1=1 crypto rx=0x7efd3400c450 tx=0x7efd3400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.864+0000 7efd2dffb640 10 client.?.objecter ms_handle_connect 0x7efd3806bc80 2023-10-30T17:55:06.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.864+0000 7efd2dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:06.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 1 -- 172.21.15.170:0/2712766467 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340074e0 con 0x7efd3806bc80 2023-10-30T17:55:06.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:06.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 10 monclient: dump: 2023-10-30T17:55:06.890 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.890 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.891 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.891 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.891 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.891 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.891 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.892 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.892 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.892 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.892 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 10 monclient: _finish_auth 0 2023-10-30T17:55:06.892 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 10 monclient: _check_auth_tickets 2023-10-30T17:55:06.892 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:06.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 1 -- 172.21.15.170:0/2712766467 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd34007640 con 0x7efd3806bc80 2023-10-30T17:55:06.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:06.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 1 -- 172.21.15.170:0/2712766467 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34004a20 con 0x7efd3806bc80 2023-10-30T17:55:06.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd3f27c640 5 monclient: authenticate success, global_id 21475 2023-10-30T17:55:06.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd3f27c640 10 monclient: _renew_subs 2023-10-30T17:55:06.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd3f27c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:06.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd380accf0 con 0x7efd3806bc80 2023-10-30T17:55:06.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:06.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:06.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2dffb640 10 monclient: dump: 2023-10-30T17:55:06.895 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:06.895 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:06.895 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.895 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:06.895 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:06.896 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:06.896 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:06.896 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:06.896 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:06.896 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:06.896 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd2f7fe640 4 set_mon_vals no callback set 2023-10-30T17:55:06.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd3f27c640 10 client.21475.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:06.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd3f27c640 10 monclient: _renew_subs 2023-10-30T17:55:06.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd3f27c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:06.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd380ae040 con 0x7efd3806bc80 2023-10-30T17:55:06.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.867+0000 7efd3f27c640 1 librados: init done 2023-10-30T17:55:06.898 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.868+0000 7efd3f27c640 10 client.21475.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:06.898 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.868+0000 7efd3f27c640 15 client.21475.objecter _session_command_op_assign -1 1 2023-10-30T17:55:06.898 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.868+0000 7efd3f27c640 20 client.21475.objecter _get_session osd=-1 returning homeless 2023-10-30T17:55:06.898 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.868+0000 7efd3f27c640 10 client.21475.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:06.898 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.868+0000 7efd3f27c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:06.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.868+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd3806cf50 con 0x7efd3806bc80 2023-10-30T17:55:06.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.885+0000 7efd2dffb640 1 -- 172.21.15.170:0/2712766467 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efd3401b360 con 0x7efd3806bc80 2023-10-30T17:55:06.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.886+0000 7efd2dffb640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd1c06c800 0x7efd1c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.886+0000 7efd3c817640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd1c06c800 0x7efd1c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.886+0000 7efd3c817640 10 monclient: get_auth_request con 0x7efd1c06c800 auth_method 0 2023-10-30T17:55:06.900 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.886+0000 7efd2dffb640 10 client.21475.objecter ms_handle_connect 0x7efd1c06c800 2023-10-30T17:55:06.900 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.886+0000 7efd3c817640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd1c06c800 0x7efd1c06ecc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7efd28000c00 tx=0x7efd2803d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.901 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 1 -- 172.21.15.170:0/2712766467 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7efd34057a90 con 0x7efd3806bc80 2023-10-30T17:55:06.901 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 10 client.21475.objecter ms_dispatch 0x7efd38000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:06.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 3 client.21475.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:06.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 3 client.21475.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:06.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 10 client.21475.objecter checking command 1 2023-10-30T17:55:06.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x7efd1c074010 0x7efd1c0764b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:06.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 20 client.21475.objecter _get_session s=0x7efd1c073940 osd=3 3 2023-10-30T17:55:06.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 20 client.21475.objecter put_session s=0x7efd1c073940 osd=3 3 2023-10-30T17:55:06.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 15 client.21475.objecter _session_command_op_remove -1 1 2023-10-30T17:55:06.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 20 client.21475.objecter _get_session s=0x7efd1c073940 osd=3 3 2023-10-30T17:55:06.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 20 client.21475.objecter get_session s=0x7efd1c073940 osd=3 3 2023-10-30T17:55:06.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 15 client.21475.objecter _session_command_op_assign 3 1 2023-10-30T17:55:06.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 20 client.21475.objecter put_session s=0x7efd1c073940 osd=3 4 2023-10-30T17:55:06.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 10 client.21475.objecter _send_command 1 2023-10-30T17:55:06.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 1 -- 172.21.15.170:0/2712766467 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd1c076b80 con 0x7efd1c074010 2023-10-30T17:55:06.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd2dffb640 20 client.21475.objecter dump_active .. 0 homeless 2023-10-30T17:55:06.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.889+0000 7efd3d819640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x7efd1c074010 0x7efd1c0764b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:06.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.890+0000 7efd3d819640 10 monclient: get_auth_request con 0x7efd1c074010 auth_method 0 2023-10-30T17:55:06.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.890+0000 7efd2dffb640 1 -- 172.21.15.170:0/2712766467 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=914) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd34053dd0 con 0x7efd3806bc80 2023-10-30T17:55:06.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.890+0000 7efd2dffb640 10 monclient: handle_get_version_reply finishing 1 version 914 2023-10-30T17:55:06.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.892+0000 7efd3d819640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x7efd1c074010 0x7efd1c0764b0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:06.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.892+0000 7efd2dffb640 10 client.21475.objecter ms_handle_connect 0x7efd1c074010 2023-10-30T17:55:06.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.914+0000 7efd2dffb640 1 -- 172.21.15.170:0/2712766467 <== osd.3 v2:172.21.15.170:6800/3657546070 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7efd1c076b80 con 0x7efd1c074010 2023-10-30T17:55:06.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.914+0000 7efd2dffb640 10 client.21475.objecter ms_dispatch 0x7efd38000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:55:06.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.914+0000 7efd2dffb640 10 client.21475.objecter _finish_command 1 = system:0 2023-10-30T17:55:06.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.914+0000 7efd2dffb640 20 client.21475.objecter put_session s=0x7efd1c073940 osd=3 4 2023-10-30T17:55:06.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.914+0000 7efd2dffb640 15 client.21475.objecter _session_command_op_remove 3 1 2023-10-30T17:55:06.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 10 client.21475.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T17:55:06.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 15 client.21475.objecter _session_command_op_assign -1 2 2023-10-30T17:55:06.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 20 client.21475.objecter _get_session s=0x7efd1c073940 osd=3 3 2023-10-30T17:55:06.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 20 client.21475.objecter put_session s=0x7efd1c073940 osd=3 3 2023-10-30T17:55:06.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 20 client.21475.objecter _get_session s=0x7efd1c073940 osd=3 3 2023-10-30T17:55:06.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 15 client.21475.objecter _session_command_op_remove -1 2 2023-10-30T17:55:06.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 20 client.21475.objecter get_session s=0x7efd1c073940 osd=3 3 2023-10-30T17:55:06.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 15 client.21475.objecter _session_command_op_assign 3 2 2023-10-30T17:55:06.930 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 20 client.21475.objecter put_session s=0x7efd1c073940 osd=3 4 2023-10-30T17:55:06.930 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 10 client.21475.objecter _send_command 2 2023-10-30T17:55:06.930 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.927+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 --> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7efd3806f610 con 0x7efd1c074010 2023-10-30T17:55:06.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd2dffb640 1 -- 172.21.15.170:0/2712766467 <== osd.3 v2:172.21.15.170:6800/3657546070 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7efd3806f610 con 0x7efd1c074010 2023-10-30T17:55:06.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd2dffb640 10 client.21475.objecter ms_dispatch 0x7efd38000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:55:06.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd2dffb640 10 client.21475.objecter _finish_command 2 = system:0 2023-10-30T17:55:06.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd2dffb640 20 client.21475.objecter put_session s=0x7efd1c073940 osd=3 4 2023-10-30T17:55:06.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd2dffb640 15 client.21475.objecter _session_command_op_remove 3 2 2023-10-30T17:55:06.943 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-30T17:55:06.943 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T17:55:06.943 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-30T17:55:06.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 10 librados: watch_flush enter 2023-10-30T17:55:06.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 10 librados: watch_flush exit 2023-10-30T17:55:06.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 10 client.21475.objecter close_session for osd.3 2023-10-30T17:55:06.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x7efd1c074010 msgr2=0x7efd1c0764b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.170:6800/3657546070,v1:172.21.15.170:6801/3657546070] conn(0x7efd1c074010 0x7efd1c0764b0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:06.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 20 client.21475.objecter put_session s=0x7efd1c073940 osd=3 1 2023-10-30T17:55:06.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 20 client.21475.objecter shutdown clearing up homeless session... 2023-10-30T17:55:06.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 10 client.21475.objecter successfully canceled tick 2023-10-30T17:55:06.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd1c06c800 msgr2=0x7efd1c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7efd1c06c800 0x7efd1c06ecc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7efd28000c00 tx=0x7efd2803d040 comp rx=0 tx=0).stop 2023-10-30T17:55:06.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 10 monclient: shutdown 2023-10-30T17:55:06.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:06.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd3806bc80 msgr2=0x7efd380adb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:06.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 1 --2- 172.21.15.170:0/2712766467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd3806bc80 0x7efd380adb00 secure :-1 s=READY pgs=5277 cs=0 l=1 rev1=1 crypto rx=0x7efd3400c450 tx=0x7efd3400c910 comp rx=0 tx=0).stop 2023-10-30T17:55:06.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3d819640 1 -- 172.21.15.170:0/2712766467 reap_dead start 2023-10-30T17:55:06.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 shutdown_connections 2023-10-30T17:55:06.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.940+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 >> 172.21.15.170:0/2712766467 conn(0x7efd380b5270 msgr2=0x7efd380b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:06.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.941+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 shutdown_connections 2023-10-30T17:55:06.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.941+0000 7efd3f27c640 1 -- 172.21.15.170:0/2712766467 wait complete. 2023-10-30T17:55:06.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:06.941+0000 7efd3f27c640 1 librados: shutdown 2023-10-30T17:55:06.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T17:55:07.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.051+0000 7f1070bb0640 10 monclient: get_monmap_and_config 2023-10-30T17:55:07.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.051+0000 7f1070bb0640 10 monclient: build_initial_monmap 2023-10-30T17:55:07.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.051+0000 7f1070bb0640 10 monclient: monmap: 2023-10-30T17:55:07.054 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:07.054 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.054 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:07.052812+0000 2023-10-30T17:55:07.054 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:07.052812+0000 2023-10-30T17:55:07.054 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:07.055 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.055 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:07.055 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:07.055 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:07.056 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:07.056 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:07.056 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.052+0000 7f1070bb0640 1 Processor -- start 2023-10-30T17:55:07.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.052+0000 7f1070bb0640 1 -- start start 2023-10-30T17:55:07.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.052+0000 7f1070bb0640 10 monclient: init 2023-10-30T17:55:07.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:07.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-10-30T17:55:07.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f106c0a8720 0x7f106c13ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 10 monclient(hunting): picked mon.noname-c con 0x7f106c0a8720 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:07.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f106c13f4d0 0x7f106c1454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 10 monclient(hunting): picked mon.noname-a con 0x7f106c13f4d0 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:07.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f106c0a8b00 0x7f106c145a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 10 monclient(hunting): picked mon.noname-b con 0x7f106c0a8b00 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:07.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c145f60 con 0x7f106c13f4d0 2023-10-30T17:55:07.062 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c0a9fd0 con 0x7f106c0a8720 2023-10-30T17:55:07.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0aa240 con 0x7f106c0a8b00 2023-10-30T17:55:07.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:07.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1070bb0640 10 monclient(hunting): authenticate will time out at 2689.376323s 2023-10-30T17:55:07.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f106a59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f106c0a8720 0x7f106c13ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f1069d9b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f106c13f4d0 0x7f106c1454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.053+0000 7f106ad9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f106c0a8b00 0x7f106c145a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106a59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f106c0a8720 0x7f106c13ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:49564/0 (socket says 172.21.15.170:49564) 2023-10-30T17:55:07.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106a59c640 1 -- 172.21.15.170:0/3219315134 learned_addr learned my addr 172.21.15.170:0/3219315134 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:07.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106a59c640 10 monclient(hunting): get_auth_request con 0x7f106c0a8720 auth_method 0 2023-10-30T17:55:07.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f1069d9b640 10 monclient(hunting): get_auth_request con 0x7f106c13f4d0 auth_method 0 2023-10-30T17:55:07.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f1069d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f1069d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f1069d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106ad9d640 10 monclient(hunting): get_auth_request con 0x7f106c0a8b00 auth_method 0 2023-10-30T17:55:07.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f1069d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f1069d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f1069d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.054+0000 7f106ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.056+0000 7f106a59c640 10 monclient(hunting): handle_auth_done global_id 21478 payload 274 2023-10-30T17:55:07.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.056+0000 7f106a59c640 1 -- 172.21.15.170:0/3219315134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f106c0a8b00 msgr2=0x7f106c145a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:07.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.056+0000 7f106a59c640 1 --2- 172.21.15.170:0/3219315134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f106c0a8b00 0x7f106c145a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.056+0000 7f106a59c640 1 -- 172.21.15.170:0/3219315134 >> v2:172.21.15.170:3300/0 conn(0x7f106c13f4d0 msgr2=0x7f106c1454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:07.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.056+0000 7f106a59c640 1 --2- 172.21.15.170:0/3219315134 >> v2:172.21.15.170:3300/0 conn(0x7f106c13f4d0 0x7f106c1454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.056+0000 7f106a59c640 10 monclient: _finish_hunting 0 2023-10-30T17:55:07.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.056+0000 7f106a59c640 1 monclient: found mon.noname-c 2023-10-30T17:55:07.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.056+0000 7f106a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:07.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.056+0000 7f106a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:55:07.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.056+0000 7f106a59c640 1 -- 172.21.15.170:0/3219315134 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c0aa520 con 0x7f106c0a8720 2023-10-30T17:55:07.071 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.058+0000 7f106a59c640 1 --2- 172.21.15.170:0/3219315134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106c0a8720 0x7f106c13ef90 secure :-1 s=READY pgs=5278 cs=0 l=1 rev1=1 crypto rx=0x7f106c067150 tx=0x7f105c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a479c8ca5cf61e4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:07.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.060+0000 7f106959a640 1 -- 172.21.15.170:0/3219315134 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c02ca70 con 0x7f106c0a8720 2023-10-30T17:55:07.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.060+0000 7f106959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.060+0000 7f106959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:07.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.060+0000 7f106959a640 10 monclient: dump: 2023-10-30T17:55:07.078 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.078 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.079 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.079 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.079 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.080 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.080 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.080 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.080 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.080 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.060+0000 7f106959a640 10 monclient: _finish_auth 0 2023-10-30T17:55:07.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.060+0000 7f106959a640 10 monclient: _check_auth_tickets 2023-10-30T17:55:07.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.060+0000 7f106959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:07.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.060+0000 7f1070bb0640 5 monclient: authenticate success, global_id 21478 2023-10-30T17:55:07.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.060+0000 7f1070bb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:07.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.061+0000 7f106959a640 1 -- 172.21.15.170:0/3219315134 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105c02cbd0 con 0x7f106c0a8720 2023-10-30T17:55:07.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.061+0000 7f106959a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:07.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.061+0000 7f106959a640 1 -- 172.21.15.170:0/3219315134 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c0343f0 con 0x7f106c0a8720 2023-10-30T17:55:07.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.061+0000 7f1070bb0640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:07.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.061+0000 7f1070bb0640 4 set_mon_vals no callback set 2023-10-30T17:55:07.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.061+0000 7f106959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.061+0000 7f106959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:07.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.061+0000 7f106959a640 10 monclient: dump: 2023-10-30T17:55:07.084 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.084 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.084 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.084 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.084 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.085 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.085 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.085 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.085 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.085 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 10 monclient: shutdown 2023-10-30T17:55:07.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:07.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 1 -- 172.21.15.170:0/3219315134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106c0a8720 msgr2=0x7f106c13ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 1 --2- 172.21.15.170:0/3219315134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106c0a8720 0x7f106c13ef90 secure :-1 s=READY pgs=5278 cs=0 l=1 rev1=1 crypto rx=0x7f106c067150 tx=0x7f105c02bed0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 1 -- 172.21.15.170:0/3219315134 shutdown_connections 2023-10-30T17:55:07.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 1 --2- 172.21.15.170:0/3219315134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f106c0a8b00 0x7f106c145a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 1 --2- 172.21.15.170:0/3219315134 >> v2:172.21.15.170:3300/0 conn(0x7f106c13f4d0 0x7f106c1454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 1 --2- 172.21.15.170:0/3219315134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106c0a8720 0x7f106c13ef90 unknown :-1 s=CLOSED pgs=5278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 1 -- 172.21.15.170:0/3219315134 >> 172.21.15.170:0/3219315134 conn(0x7f106c0b7300 msgr2=0x7f106c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:07.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 1 -- 172.21.15.170:0/3219315134 shutdown_connections 2023-10-30T17:55:07.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.062+0000 7f1070bb0640 1 -- 172.21.15.170:0/3219315134 wait complete. 2023-10-30T17:55:07.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.063+0000 7f1070bb0640 10 monclient: build_initial_monmap 2023-10-30T17:55:07.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.063+0000 7f1070bb0640 10 monclient: monmap: 2023-10-30T17:55:07.088 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:07.089 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:07.089 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:07.089 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:07.089 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:07.089 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.090 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:07.090 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:07.090 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:07.090 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.063+0000 7f1070bb0640 1 librados: starting msgr at 2023-10-30T17:55:07.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.063+0000 7f1070bb0640 1 librados: starting objecter 2023-10-30T17:55:07.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 1 Processor -- start 2023-10-30T17:55:07.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 1 -- start start 2023-10-30T17:55:07.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 1 librados: setting wanted keys 2023-10-30T17:55:07.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 1 librados: calling monclient init 2023-10-30T17:55:07.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient: init 2023-10-30T17:55:07.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:07.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:55:07.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f106c0a8720 0x7f106c0ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient(hunting): picked mon.noname-a con 0x7f106c0a8720 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:07.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f106c0a8b00 0x7f106c0b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient(hunting): picked mon.noname-c con 0x7f106c0a8b00 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:07.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106c13f4d0 0x7f106c145230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient(hunting): picked mon.noname-b con 0x7f106c13f4d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:07.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0adb00 con 0x7f106c0a8720 2023-10-30T17:55:07.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0b12a0 con 0x7f106c13f4d0 2023-10-30T17:55:07.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0b0d80 con 0x7f106c0a8b00 2023-10-30T17:55:07.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:07.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.064+0000 7f1070bb0640 10 monclient(hunting): authenticate will time out at 2689.387676s 2023-10-30T17:55:07.095 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106a59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f106c0a8720 0x7f106c0ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f1069d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f106c0a8b00 0x7f106c0b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106c13f4d0 0x7f106c145230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106a59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f106c0a8720 0x7f106c0ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40502/0 (socket says 172.21.15.170:40502) 2023-10-30T17:55:07.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106a59c640 1 -- 172.21.15.170:0/1507674520 learned_addr learned my addr 172.21.15.170:0/1507674520 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:07.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106a59c640 10 monclient(hunting): get_auth_request con 0x7f106c0a8720 auth_method 0 2023-10-30T17:55:07.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106ad9d640 10 monclient(hunting): get_auth_request con 0x7f106c13f4d0 auth_method 0 2023-10-30T17:55:07.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f1069d9b640 10 monclient(hunting): get_auth_request con 0x7f106c0a8b00 auth_method 0 2023-10-30T17:55:07.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f1069d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f1069d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f1069d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.065+0000 7f106ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.066+0000 7f1069d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.066+0000 7f1069d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.066+0000 7f1069d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.068+0000 7f106ad9d640 10 monclient(hunting): handle_auth_done global_id 21481 payload 824 2023-10-30T17:55:07.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.068+0000 7f106ad9d640 1 -- 172.21.15.170:0/1507674520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f106c0a8b00 msgr2=0x7f106c0b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.068+0000 7f106ad9d640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f106c0a8b00 0x7f106c0b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.068+0000 7f106ad9d640 1 -- 172.21.15.170:0/1507674520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f106c0a8720 msgr2=0x7f106c0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.068+0000 7f106ad9d640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f106c0a8720 0x7f106c0ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.068+0000 7f106ad9d640 10 monclient: _finish_hunting 0 2023-10-30T17:55:07.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.068+0000 7f106ad9d640 1 monclient: found mon.noname-b 2023-10-30T17:55:07.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.068+0000 7f106ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:07.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.068+0000 7f106ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:07.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.068+0000 7f106ad9d640 1 -- 172.21.15.170:0/1507674520 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c009b30 con 0x7f106c13f4d0 2023-10-30T17:55:07.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.070+0000 7f106ad9d640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106c13f4d0 0x7f106c145230 secure :-1 s=READY pgs=5279 cs=0 l=1 rev1=1 crypto rx=0x7f106400e7f0 tx=0x7f106400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:07.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.070+0000 7f105affd640 10 client.?.objecter ms_handle_connect 0x7f106c13f4d0 2023-10-30T17:55:07.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.070+0000 7f105affd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:07.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 1 -- 172.21.15.170:0/1507674520 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064009590 con 0x7f106c13f4d0 2023-10-30T17:55:07.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:07.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 10 monclient: dump: 2023-10-30T17:55:07.104 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.105 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.105 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.105 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.105 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.106 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.106 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.106 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.106 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.106 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 10 monclient: _finish_auth 0 2023-10-30T17:55:07.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 10 monclient: _check_auth_tickets 2023-10-30T17:55:07.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:07.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 1 -- 172.21.15.170:0/1507674520 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106400c940 con 0x7f106c13f4d0 2023-10-30T17:55:07.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f1070bb0640 5 monclient: authenticate success, global_id 21481 2023-10-30T17:55:07.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f1070bb0640 10 monclient: _renew_subs 2023-10-30T17:55:07.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f1070bb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:07.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f1070bb0640 1 -- 172.21.15.170:0/1507674520 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c0b01f0 con 0x7f106c13f4d0 2023-10-30T17:55:07.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:07.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 1 -- 172.21.15.170:0/1507674520 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106400ccd0 con 0x7f106c13f4d0 2023-10-30T17:55:07.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:07.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f105affd640 10 monclient: dump: 2023-10-30T17:55:07.109 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.109 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.110 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.110 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.110 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.110 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.111 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.111 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.111 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.111 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f1068d99640 4 set_mon_vals no callback set 2023-10-30T17:55:07.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f1070bb0640 10 client.21481.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:07.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f1070bb0640 10 monclient: _renew_subs 2023-10-30T17:55:07.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f1070bb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:07.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f1070bb0640 1 -- 172.21.15.170:0/1507674520 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c145950 con 0x7f106c13f4d0 2023-10-30T17:55:07.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.074+0000 7f1070bb0640 1 librados: init done 2023-10-30T17:55:07.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.075+0000 7f1070bb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:07.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.075+0000 7f1070bb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:07.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.075+0000 7f1070bb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:07.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.075+0000 7f1070bb0640 1 -- 172.21.15.170:0/1507674520 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c0b0fa0 con 0x7f106c13f4d0 2023-10-30T17:55:07.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.315+0000 7f105affd640 1 -- 172.21.15.170:0/1507674520 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1064004020 con 0x7f106c13f4d0 2023-10-30T17:55:07.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.315+0000 7f105affd640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f104c06c800 0x7f104c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.315+0000 7f106a59c640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f104c06c800 0x7f104c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.315+0000 7f106a59c640 10 monclient: get_auth_request con 0x7f104c06c800 auth_method 0 2023-10-30T17:55:07.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.316+0000 7f106a59c640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f104c06c800 0x7f104c06ecc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f105c000f80 tx=0x7f105c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:07.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.316+0000 7f105affd640 10 client.21481.objecter ms_handle_connect 0x7f104c06c800 2023-10-30T17:55:07.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.319+0000 7f105affd640 1 -- 172.21.15.170:0/1507674520 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f1064058490 con 0x7f106c13f4d0 2023-10-30T17:55:07.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.319+0000 7f105affd640 10 client.21481.objecter ms_dispatch 0x7f106c000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:07.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.319+0000 7f105affd640 3 client.21481.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:07.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.319+0000 7f105affd640 3 client.21481.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:07.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.319+0000 7f105affd640 20 client.21481.objecter dump_active .. 0 homeless 2023-10-30T17:55:07.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.329+0000 7f105affd640 1 -- 172.21.15.170:0/1507674520 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1064058260 con 0x7f106c13f4d0 2023-10-30T17:55:07.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.329+0000 7f105affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:07.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.329+0000 7f105affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:07.499 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.498+0000 7f1070bb0640 1 -- 172.21.15.170:0/1507674520 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f106c0b0520 con 0x7f104c06c800 2023-10-30T17:55:07.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.499+0000 7f105affd640 1 -- 172.21.15.170:0/1507674520 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f106c0b0520 con 0x7f104c06c800 2023-10-30T17:55:07.500 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T17:55:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 10 librados: watch_flush enter 2023-10-30T17:55:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 10 librados: watch_flush exit 2023-10-30T17:55:07.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 20 client.21481.objecter shutdown clearing up homeless session... 2023-10-30T17:55:07.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 10 client.21481.objecter successfully canceled tick 2023-10-30T17:55:07.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 -- 172.21.15.170:0/1507674520 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f104c06c800 msgr2=0x7f104c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f104c06c800 0x7f104c06ecc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f105c000f80 tx=0x7f105c030000 comp rx=0 tx=0).stop 2023-10-30T17:55:07.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 10 monclient: shutdown 2023-10-30T17:55:07.504 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 -- 172.21.15.170:0/1507674520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106c13f4d0 msgr2=0x7f106c145230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106c13f4d0 0x7f106c145230 secure :-1 s=READY pgs=5279 cs=0 l=1 rev1=1 crypto rx=0x7f106400e7f0 tx=0x7f106400ecb0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 -- 172.21.15.170:0/1507674520 shutdown_connections 2023-10-30T17:55:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f104c06c800 0x7f104c06ecc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106c13f4d0 0x7f106c145230 unknown :-1 s=CLOSED pgs=5279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f106c0a8b00 0x7f106c0b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 --2- 172.21.15.170:0/1507674520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f106c0a8720 0x7f106c0ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 -- 172.21.15.170:0/1507674520 >> 172.21.15.170:0/1507674520 conn(0x7f106c0b7300 msgr2=0x7f106c138c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:07.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 -- 172.21.15.170:0/1507674520 shutdown_connections 2023-10-30T17:55:07.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 -- 172.21.15.170:0/1507674520 wait complete. 2023-10-30T17:55:07.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.501+0000 7f1070bb0640 1 librados: shutdown 2023-10-30T17:55:07.514 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T17:55:07.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T17:55:07.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.620+0000 7fde03725640 10 monclient: get_monmap_and_config 2023-10-30T17:55:07.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.620+0000 7fde03725640 10 monclient: build_initial_monmap 2023-10-30T17:55:07.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.620+0000 7fde03725640 10 monclient: monmap: 2023-10-30T17:55:07.622 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:07.622 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.622 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:07.621466+0000 2023-10-30T17:55:07.623 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:07.621466+0000 2023-10-30T17:55:07.623 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:07.623 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.623 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:07.623 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:07.624 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:07.624 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:07.624 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:07.624 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 1 Processor -- start 2023-10-30T17:55:07.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 1 -- start start 2023-10-30T17:55:07.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient: init 2023-10-30T17:55:07.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:07.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-10-30T17:55:07.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fddfc0b3e20 0x7fddfc0a8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient(hunting): picked mon.noname-a con 0x7fddfc0b3e20 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:07.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fddfc144fa0 0x7fddfc13ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient(hunting): picked mon.noname-c con 0x7fddfc144fa0 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:07.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fddfc13efd0 0x7fddfc0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient(hunting): picked mon.noname-b con 0x7fddfc13efd0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:07.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddfc0b4200 con 0x7fddfc0b3e20 2023-10-30T17:55:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddfc0aa950 con 0x7fddfc144fa0 2023-10-30T17:55:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc0a8ba0 con 0x7fddfc13efd0 2023-10-30T17:55:07.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:07.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.621+0000 7fde03725640 10 monclient(hunting): authenticate will time out at 2689.944961s 2023-10-30T17:55:07.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.624+0000 7fde00cc0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fddfc144fa0 0x7fddfc13ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.624+0000 7fde00cc0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fddfc144fa0 0x7fddfc13ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:49582/0 (socket says 172.21.15.170:49582) 2023-10-30T17:55:07.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.624+0000 7fde00cc0640 1 -- 172.21.15.170:0/161459541 learned_addr learned my addr 172.21.15.170:0/161459541 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:07.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.624+0000 7fde00cc0640 10 monclient(hunting): get_auth_request con 0x7fddfc144fa0 auth_method 0 2023-10-30T17:55:07.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.624+0000 7fde00cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.624+0000 7fde00cc0640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.624+0000 7fde00cc0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.624+0000 7fde00cc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.624+0000 7fde00cc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.624+0000 7fde00cc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.626+0000 7fde00cc0640 10 monclient(hunting): handle_auth_done global_id 21484 payload 274 2023-10-30T17:55:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.626+0000 7fde00cc0640 1 -- 172.21.15.170:0/161459541 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fddfc13efd0 msgr2=0x7fddfc0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.626+0000 7fde00cc0640 1 --2- 172.21.15.170:0/161459541 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fddfc13efd0 0x7fddfc0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.626+0000 7fde00cc0640 1 -- 172.21.15.170:0/161459541 >> v2:172.21.15.170:3300/0 conn(0x7fddfc0b3e20 msgr2=0x7fddfc0a8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.626+0000 7fde00cc0640 1 --2- 172.21.15.170:0/161459541 >> v2:172.21.15.170:3300/0 conn(0x7fddfc0b3e20 0x7fddfc0a8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.626+0000 7fde00cc0640 10 monclient: _finish_hunting 0 2023-10-30T17:55:07.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.626+0000 7fde00cc0640 1 monclient: found mon.noname-c 2023-10-30T17:55:07.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.626+0000 7fde00cc0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:07.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.626+0000 7fde00cc0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:55:07.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.626+0000 7fde00cc0640 1 -- 172.21.15.170:0/161459541 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc0aa320 con 0x7fddfc144fa0 2023-10-30T17:55:07.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.628+0000 7fde00cc0640 1 --2- 172.21.15.170:0/161459541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc144fa0 0x7fddfc13ea90 secure :-1 s=READY pgs=5280 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0a9c10 tx=0x7fddec02b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=662e6be84aa49a1e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 1 -- 172.21.15.170:0/161459541 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec031750 con 0x7fddfc144fa0 2023-10-30T17:55:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 10 monclient: dump: 2023-10-30T17:55:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.638 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.638 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.638 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.638 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.638 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.639 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 10 monclient: _finish_auth 0 2023-10-30T17:55:07.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 10 monclient: _check_auth_tickets 2023-10-30T17:55:07.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:07.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 1 -- 172.21.15.170:0/161459541 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddec0318b0 con 0x7fddfc144fa0 2023-10-30T17:55:07.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:07.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 1 -- 172.21.15.170:0/161459541 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec031c40 con 0x7fddfc144fa0 2023-10-30T17:55:07.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fde03725640 5 monclient: authenticate success, global_id 21484 2023-10-30T17:55:07.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fde03725640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:07.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fde03725640 4 set_mon_vals no callback set 2023-10-30T17:55:07.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:07.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.634+0000 7fddf3fff640 10 monclient: dump: 2023-10-30T17:55:07.641 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.641 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.641 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.642 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.642 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.642 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.642 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.642 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.643 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.643 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 10 monclient: shutdown 2023-10-30T17:55:07.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:07.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 1 -- 172.21.15.170:0/161459541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc144fa0 msgr2=0x7fddfc13ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 1 --2- 172.21.15.170:0/161459541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc144fa0 0x7fddfc13ea90 secure :-1 s=READY pgs=5280 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0a9c10 tx=0x7fddec02b7f0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 1 -- 172.21.15.170:0/161459541 shutdown_connections 2023-10-30T17:55:07.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 1 --2- 172.21.15.170:0/161459541 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fddfc13efd0 0x7fddfc0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 1 --2- 172.21.15.170:0/161459541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc144fa0 0x7fddfc13ea90 unknown :-1 s=CLOSED pgs=5280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 1 --2- 172.21.15.170:0/161459541 >> v2:172.21.15.170:3300/0 conn(0x7fddfc0b3e20 0x7fddfc0a8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 1 -- 172.21.15.170:0/161459541 >> 172.21.15.170:0/161459541 conn(0x7fddfc071770 msgr2=0x7fddfc071b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:07.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 1 -- 172.21.15.170:0/161459541 shutdown_connections 2023-10-30T17:55:07.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.635+0000 7fde03725640 1 -- 172.21.15.170:0/161459541 wait complete. 2023-10-30T17:55:07.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.636+0000 7fde03725640 10 monclient: build_initial_monmap 2023-10-30T17:55:07.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.636+0000 7fde03725640 10 monclient: monmap: 2023-10-30T17:55:07.648 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:07.649 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:07.649 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:07.649 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:07.649 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:07.649 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.650 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:07.650 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:07.650 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:07.650 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.650 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.636+0000 7fde03725640 1 librados: starting msgr at 2023-10-30T17:55:07.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.636+0000 7fde03725640 1 librados: starting objecter 2023-10-30T17:55:07.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.636+0000 7fde03725640 1 Processor -- start 2023-10-30T17:55:07.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.636+0000 7fde03725640 1 -- start start 2023-10-30T17:55:07.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.636+0000 7fde03725640 1 librados: setting wanted keys 2023-10-30T17:55:07.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.636+0000 7fde03725640 1 librados: calling monclient init 2023-10-30T17:55:07.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.636+0000 7fde03725640 10 monclient: init 2023-10-30T17:55:07.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:07.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:55:07.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fddfc0b3e20 0x7fddfc1d6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 10 monclient(hunting): picked mon.noname-c con 0x7fddfc0b3e20 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:07.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc13efd0 0x7fddfc13b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 10 monclient(hunting): picked mon.noname-b con 0x7fddfc13efd0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:07.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddfc144fa0 0x7fddfc13b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 10 monclient(hunting): picked mon.noname-a con 0x7fddfc144fa0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:07.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc141870 con 0x7fddfc144fa0 2023-10-30T17:55:07.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc1d7470 con 0x7fddfc13efd0 2023-10-30T17:55:07.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc1d6a10 con 0x7fddfc0b3e20 2023-10-30T17:55:07.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:07.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde03725640 10 monclient(hunting): authenticate will time out at 2689.960280s 2023-10-30T17:55:07.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde014c1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fddfc0b3e20 0x7fddfc1d6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.637+0000 7fde01cc2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddfc144fa0 0x7fddfc13b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde014c1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fddfc0b3e20 0x7fddfc1d6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:39070/0 (socket says 172.21.15.170:39070) 2023-10-30T17:55:07.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde014c1640 1 -- 172.21.15.170:0/2120761511 learned_addr learned my addr 172.21.15.170:0/2120761511 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:07.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde014c1640 10 monclient(hunting): get_auth_request con 0x7fddfc0b3e20 auth_method 0 2023-10-30T17:55:07.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde014c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde014c1640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde014c1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde01cc2640 10 monclient(hunting): get_auth_request con 0x7fddfc144fa0 auth_method 0 2023-10-30T17:55:07.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde01cc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde01cc2640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde01cc2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde014c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde014c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde014c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.638+0000 7fde00cc0640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc13efd0 0x7fddfc13b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.639+0000 7fde00cc0640 10 monclient(hunting): get_auth_request con 0x7fddfc13efd0 auth_method 0 2023-10-30T17:55:07.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.639+0000 7fde00cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.639+0000 7fde00cc0640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.639+0000 7fde00cc0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.639+0000 7fde00cc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.639+0000 7fde00cc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.639+0000 7fde00cc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.640+0000 7fde01cc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.640+0000 7fde01cc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.640+0000 7fde01cc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.641+0000 7fde014c1640 10 monclient(hunting): handle_auth_done global_id 16349 payload 824 2023-10-30T17:55:07.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.641+0000 7fde014c1640 1 -- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc13efd0 msgr2=0x7fddfc13b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.641+0000 7fde014c1640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc13efd0 0x7fddfc13b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.641+0000 7fde014c1640 1 -- 172.21.15.170:0/2120761511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddfc144fa0 msgr2=0x7fddfc13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.641+0000 7fde014c1640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddfc144fa0 0x7fddfc13b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.641+0000 7fde014c1640 10 monclient: _finish_hunting 0 2023-10-30T17:55:07.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.641+0000 7fde014c1640 1 monclient: found mon.noname-c 2023-10-30T17:55:07.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.641+0000 7fde014c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:07.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.641+0000 7fde014c1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:55:07.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.641+0000 7fde014c1640 1 -- 172.21.15.170:0/2120761511 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec0094d0 con 0x7fddfc0b3e20 2023-10-30T17:55:07.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.644+0000 7fde014c1640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fddfc0b3e20 0x7fddfc1d6f30 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7fddf800b4a0 tx=0x7fddf800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:07.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.644+0000 7fddf1ffb640 10 client.?.objecter ms_handle_connect 0x7fddfc0b3e20 2023-10-30T17:55:07.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.644+0000 7fddf1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:07.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 1 -- 172.21.15.170:0/2120761511 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8010610 con 0x7fddfc0b3e20 2023-10-30T17:55:07.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:07.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 10 monclient: dump: 2023-10-30T17:55:07.664 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.665 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.665 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.665 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.665 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.665 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.665 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.666 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.666 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.666 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 10 monclient: _finish_auth 0 2023-10-30T17:55:07.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:55:07.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:07.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 1 -- 172.21.15.170:0/2120761511 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf8010770 con 0x7fddfc0b3e20 2023-10-30T17:55:07.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:07.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fde03725640 5 monclient: authenticate success, global_id 16349 2023-10-30T17:55:07.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 1 -- 172.21.15.170:0/2120761511 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8010b20 con 0x7fddfc0b3e20 2023-10-30T17:55:07.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:55:07.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf1ffb640 10 monclient: dump: 2023-10-30T17:55:07.668 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.668 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.669 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.669 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.669 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.669 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.669 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.670 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.671 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.671 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fde03725640 10 monclient: _renew_subs 2023-10-30T17:55:07.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fde03725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:55:07.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fddf37fe640 4 set_mon_vals no callback set 2023-10-30T17:55:07.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddfc1d65c0 con 0x7fddfc0b3e20 2023-10-30T17:55:07.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fde03725640 10 client.16349.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:07.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fde03725640 10 monclient: _renew_subs 2023-10-30T17:55:07.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fde03725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:55:07.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddfc13c050 con 0x7fddfc0b3e20 2023-10-30T17:55:07.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.648+0000 7fde03725640 1 librados: init done 2023-10-30T17:55:07.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.649+0000 7fde03725640 10 client.16349.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:07.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.649+0000 7fde03725640 15 client.16349.objecter _session_command_op_assign -1 1 2023-10-30T17:55:07.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.649+0000 7fde03725640 20 client.16349.objecter _get_session osd=-1 returning homeless 2023-10-30T17:55:07.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.649+0000 7fde03725640 10 client.16349.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:07.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.649+0000 7fde03725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:55:07.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.649+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fddfc13f560 con 0x7fddfc0b3e20 2023-10-30T17:55:07.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.668+0000 7fddf1ffb640 1 -- 172.21.15.170:0/2120761511 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fddf8010c80 con 0x7fddfc0b3e20 2023-10-30T17:55:07.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.668+0000 7fddf1ffb640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fdde006c850 0x7fdde006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.668+0000 7fde00cc0640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fdde006c850 0x7fdde006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.669+0000 7fde00cc0640 10 monclient: get_auth_request con 0x7fdde006c850 auth_method 0 2023-10-30T17:55:07.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.669+0000 7fde00cc0640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fdde006c850 0x7fdde006ed10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fddec035000 tx=0x7fddec008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:07.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.669+0000 7fddf1ffb640 10 client.16349.objecter ms_handle_connect 0x7fdde006c850 2023-10-30T17:55:07.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 1 -- 172.21.15.170:0/2120761511 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fddf80585a0 con 0x7fddfc0b3e20 2023-10-30T17:55:07.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 10 client.16349.objecter ms_dispatch 0x7fddfc000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:07.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 3 client.16349.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:07.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 3 client.16349.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:07.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 10 client.16349.objecter checking command 1 2023-10-30T17:55:07.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] conn(0x7fdde0074060 0x7fdde0076500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 20 client.16349.objecter _get_session s=0x7fdde0073990 osd=4 3 2023-10-30T17:55:07.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 20 client.16349.objecter put_session s=0x7fdde0073990 osd=4 3 2023-10-30T17:55:07.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 15 client.16349.objecter _session_command_op_remove -1 1 2023-10-30T17:55:07.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 20 client.16349.objecter _get_session s=0x7fdde0073990 osd=4 3 2023-10-30T17:55:07.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 20 client.16349.objecter get_session s=0x7fdde0073990 osd=4 3 2023-10-30T17:55:07.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 15 client.16349.objecter _session_command_op_assign 4 1 2023-10-30T17:55:07.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 20 client.16349.objecter put_session s=0x7fdde0073990 osd=4 4 2023-10-30T17:55:07.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 10 client.16349.objecter _send_command 1 2023-10-30T17:55:07.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 1 -- 172.21.15.170:0/2120761511 --> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdde0076bd0 con 0x7fdde0074060 2023-10-30T17:55:07.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.671+0000 7fddf1ffb640 20 client.16349.objecter dump_active .. 0 homeless 2023-10-30T17:55:07.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.672+0000 7fde01cc2640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] conn(0x7fdde0074060 0x7fdde0076500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.672+0000 7fde01cc2640 10 monclient: get_auth_request con 0x7fdde0074060 auth_method 0 2023-10-30T17:55:07.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.675+0000 7fde01cc2640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] conn(0x7fdde0074060 0x7fdde0076500 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:07.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.675+0000 7fddf1ffb640 10 client.16349.objecter ms_handle_connect 0x7fdde0074060 2023-10-30T17:55:07.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.697+0000 7fddf1ffb640 1 -- 172.21.15.170:0/2120761511 <== osd.4 v2:172.21.15.177:6802/2107882913 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fdde0076bd0 con 0x7fdde0074060 2023-10-30T17:55:07.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.697+0000 7fddf1ffb640 10 client.16349.objecter ms_dispatch 0x7fddfc000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:55:07.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.698+0000 7fddf1ffb640 10 client.16349.objecter _finish_command 1 = system:0 2023-10-30T17:55:07.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.698+0000 7fddf1ffb640 20 client.16349.objecter put_session s=0x7fdde0073990 osd=4 4 2023-10-30T17:55:07.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.698+0000 7fddf1ffb640 15 client.16349.objecter _session_command_op_remove 4 1 2023-10-30T17:55:07.712 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 10 client.16349.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T17:55:07.712 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 15 client.16349.objecter _session_command_op_assign -1 2 2023-10-30T17:55:07.712 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 20 client.16349.objecter _get_session s=0x7fdde0073990 osd=4 3 2023-10-30T17:55:07.712 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 20 client.16349.objecter put_session s=0x7fdde0073990 osd=4 3 2023-10-30T17:55:07.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 20 client.16349.objecter _get_session s=0x7fdde0073990 osd=4 3 2023-10-30T17:55:07.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 15 client.16349.objecter _session_command_op_remove -1 2 2023-10-30T17:55:07.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 20 client.16349.objecter get_session s=0x7fdde0073990 osd=4 3 2023-10-30T17:55:07.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 15 client.16349.objecter _session_command_op_assign 4 2 2023-10-30T17:55:07.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 20 client.16349.objecter put_session s=0x7fdde0073990 osd=4 4 2023-10-30T17:55:07.714 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 10 client.16349.objecter _send_command 2 2023-10-30T17:55:07.714 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.711+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 --> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fddfc0b3460 con 0x7fdde0074060 2023-10-30T17:55:07.726 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fddf1ffb640 1 -- 172.21.15.170:0/2120761511 <== osd.4 v2:172.21.15.177:6802/2107882913 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fddf400a4e0 con 0x7fdde0074060 2023-10-30T17:55:07.726 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fddf1ffb640 10 client.16349.objecter ms_dispatch 0x7fddfc000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:55:07.726 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fddf1ffb640 10 client.16349.objecter _finish_command 2 = system:0 2023-10-30T17:55:07.727 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fddf1ffb640 20 client.16349.objecter put_session s=0x7fdde0073990 osd=4 4 2023-10-30T17:55:07.727 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fddf1ffb640 15 client.16349.objecter _session_command_op_remove 4 2 2023-10-30T17:55:07.727 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-30T17:55:07.727 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T17:55:07.728 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-30T17:55:07.729 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 10 librados: watch_flush enter 2023-10-30T17:55:07.729 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 10 librados: watch_flush exit 2023-10-30T17:55:07.729 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 10 client.16349.objecter close_session for osd.4 2023-10-30T17:55:07.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] conn(0x7fdde0074060 msgr2=0x7fdde0076500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:6802/2107882913,v1:172.21.15.177:6803/2107882913] conn(0x7fdde0074060 0x7fdde0076500 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 20 client.16349.objecter put_session s=0x7fdde0073990 osd=4 1 2023-10-30T17:55:07.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 20 client.16349.objecter shutdown clearing up homeless session... 2023-10-30T17:55:07.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 10 client.16349.objecter successfully canceled tick 2023-10-30T17:55:07.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fdde006c850 msgr2=0x7fdde006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fdde006c850 0x7fdde006ed10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fddec035000 tx=0x7fddec008030 comp rx=0 tx=0).stop 2023-10-30T17:55:07.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 10 monclient: shutdown 2023-10-30T17:55:07.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 20 monclient: shutdown canceling and discarding version request 1 2023-10-30T17:55:07.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:07.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fddfc0b3e20 msgr2=0x7fddfc1d6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 --2- 172.21.15.170:0/2120761511 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fddfc0b3e20 0x7fddfc1d6f30 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7fddf800b4a0 tx=0x7fddf800b960 comp rx=0 tx=0).stop 2023-10-30T17:55:07.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde01cc2640 1 -- 172.21.15.170:0/2120761511 reap_dead start 2023-10-30T17:55:07.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 shutdown_connections 2023-10-30T17:55:07.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 >> 172.21.15.170:0/2120761511 conn(0x7fddfc071770 msgr2=0x7fddfc13ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:07.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 shutdown_connections 2023-10-30T17:55:07.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 -- 172.21.15.170:0/2120761511 wait complete. 2023-10-30T17:55:07.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.725+0000 7fde03725640 1 librados: shutdown 2023-10-30T17:55:07.737 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T17:55:07.837 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.835+0000 7f0ace5b0640 10 monclient: get_monmap_and_config 2023-10-30T17:55:07.837 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.835+0000 7f0ace5b0640 10 monclient: build_initial_monmap 2023-10-30T17:55:07.837 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.835+0000 7f0ace5b0640 10 monclient: monmap: 2023-10-30T17:55:07.838 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:07.838 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.838 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:07.836702+0000 2023-10-30T17:55:07.838 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:07.836702+0000 2023-10-30T17:55:07.838 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:07.839 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.839 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:07.839 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:07.839 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:07.839 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:07.839 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:07.840 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.836+0000 7f0ace5b0640 1 Processor -- start 2023-10-30T17:55:07.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.836+0000 7f0ace5b0640 1 -- start start 2023-10-30T17:55:07.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.836+0000 7f0ace5b0640 10 monclient: init 2023-10-30T17:55:07.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:07.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-10-30T17:55:07.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ac80b0ec0 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:07.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0ac80aef50 0x7f0ac8071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 10 monclient(hunting): picked mon.noname-a con 0x7f0ac80aef50 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:07.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ac813cd30 0x7f0ac814a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 10 monclient(hunting): picked mon.noname-b con 0x7f0ac813cd30 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:07.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac8070de0 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.845 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.845 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac8143550 con 0x7f0ac80aef50 2023-10-30T17:55:07.845 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.845 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac80b12a0 con 0x7f0ac813cd30 2023-10-30T17:55:07.845 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:07.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ace5b0640 10 monclient(hunting): authenticate will time out at 2690.160352s 2023-10-30T17:55:07.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ac7fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f0ac80b0ec0 0x7f0ac8142e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:45934/0 (socket says 172.21.15.170:45934) 2023-10-30T17:55:07.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ac7fff640 1 -- 172.21.15.170:0/32576833 learned_addr learned my addr 172.21.15.170:0/32576833 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:07.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0ac77fe640 1 --2- 172.21.15.170:0/32576833 >> v2:172.21.15.170:3300/0 conn(0x7f0ac80aef50 0x7f0ac8071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.837+0000 7f0accb4d640 1 --2- 172.21.15.170:0/32576833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ac813cd30 0x7f0ac814a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0ac77fe640 10 monclient(hunting): get_auth_request con 0x7f0ac80aef50 auth_method 0 2023-10-30T17:55:07.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0ac77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0ac77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0ac77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0accb4d640 10 monclient(hunting): get_auth_request con 0x7f0ac813cd30 auth_method 0 2023-10-30T17:55:07.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0accb4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0accb4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0accb4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0ac77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0ac77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0ac77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0accb4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0accb4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.838+0000 7f0accb4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac6ffd640 1 -- 172.21.15.170:0/32576833 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607371745 0 0) 0x7f0ac8070de0 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac6ffd640 10 monclient(hunting): _negotiate 2023-10-30T17:55:07.850 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.850 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.850 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac6ffd640 10 monclient(hunting): my global_id is 21490 2023-10-30T17:55:07.850 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac6ffd640 1 -- 172.21.15.170:0/32576833 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab0003520 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.850 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac77fe640 10 monclient(hunting): handle_auth_done global_id 16044 payload 274 2023-10-30T17:55:07.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac77fe640 1 -- 172.21.15.170:0/32576833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ac813cd30 msgr2=0x7f0ac814a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:07.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac77fe640 1 --2- 172.21.15.170:0/32576833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ac813cd30 0x7f0ac814a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac77fe640 1 -- 172.21.15.170:0/32576833 >> v1:172.21.15.177:6789/0 conn(0x7f0ac80b0ec0 legacy=0x7f0ac8142e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac77fe640 10 monclient: _finish_hunting 0 2023-10-30T17:55:07.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac77fe640 1 monclient: found mon.noname-a 2023-10-30T17:55:07.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:07.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:07.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.840+0000 7f0ac77fe640 1 -- 172.21.15.170:0/32576833 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac80722b0 con 0x7f0ac80aef50 2023-10-30T17:55:07.856 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.842+0000 7f0ac77fe640 1 --2- 172.21.15.170:0/32576833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ac80aef50 0x7f0ac8071270 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097d0 tx=0x7f0abc02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479d38ff279e46c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:07.856 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 1 -- 172.21.15.170:0/32576833 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc031770 con 0x7f0ac80aef50 2023-10-30T17:55:07.856 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.857 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:07.857 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 10 monclient: dump: 2023-10-30T17:55:07.857 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.857 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.857 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.858 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.858 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.858 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.858 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.858 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.859 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.859 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.859 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 10 monclient: _finish_auth 0 2023-10-30T17:55:07.859 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:55:07.859 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:07.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 1 -- 172.21.15.170:0/32576833 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc0318d0 con 0x7f0ac80aef50 2023-10-30T17:55:07.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:07.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 1 -- 172.21.15.170:0/32576833 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc031c60 con 0x7f0ac80aef50 2023-10-30T17:55:07.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:07.861 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ac6ffd640 10 monclient: dump: 2023-10-30T17:55:07.861 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.861 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.861 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.861 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.862 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.862 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.862 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.862 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.862 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.862 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ace5b0640 5 monclient: authenticate success, global_id 16044 2023-10-30T17:55:07.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ace5b0640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:07.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.845+0000 7f0ace5b0640 4 set_mon_vals no callback set 2023-10-30T17:55:07.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.846+0000 7f0ace5b0640 10 monclient: shutdown 2023-10-30T17:55:07.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.846+0000 7f0ace5b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:07.864 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.846+0000 7f0ace5b0640 1 -- 172.21.15.170:0/32576833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ac80aef50 msgr2=0x7f0ac8071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.864 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.846+0000 7f0ace5b0640 1 --2- 172.21.15.170:0/32576833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ac80aef50 0x7f0ac8071270 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097d0 tx=0x7f0abc02b810 comp rx=0 tx=0).stop 2023-10-30T17:55:07.864 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.846+0000 7f0ace5b0640 1 -- 172.21.15.170:0/32576833 shutdown_connections 2023-10-30T17:55:07.864 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.846+0000 7f0ace5b0640 1 --2- 172.21.15.170:0/32576833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ac813cd30 0x7f0ac814a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.864 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.846+0000 7f0ace5b0640 1 --2- 172.21.15.170:0/32576833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ac80aef50 0x7f0ac8071270 unknown :-1 s=CLOSED pgs=3375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.846+0000 7f0ace5b0640 1 -- 172.21.15.170:0/32576833 >> 172.21.15.170:0/32576833 conn(0x7f0ac80b1d40 msgr2=0x7f0ac80b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:07.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.846+0000 7f0ace5b0640 1 -- 172.21.15.170:0/32576833 shutdown_connections 2023-10-30T17:55:07.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.846+0000 7f0ace5b0640 1 -- 172.21.15.170:0/32576833 wait complete. 2023-10-30T17:55:07.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.847+0000 7f0ace5b0640 10 monclient: build_initial_monmap 2023-10-30T17:55:07.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.847+0000 7f0ace5b0640 10 monclient: monmap: 2023-10-30T17:55:07.865 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:07.866 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:07.866 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:07.866 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:07.866 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:07.866 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.867 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:07.867 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:07.867 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:07.867 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.847+0000 7f0ace5b0640 1 librados: starting msgr at 2023-10-30T17:55:07.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.847+0000 7f0ace5b0640 1 librados: starting objecter 2023-10-30T17:55:07.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.847+0000 7f0ace5b0640 1 Processor -- start 2023-10-30T17:55:07.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 1 -- start start 2023-10-30T17:55:07.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 1 librados: setting wanted keys 2023-10-30T17:55:07.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 1 librados: calling monclient init 2023-10-30T17:55:07.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient: init 2023-10-30T17:55:07.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:07.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T17:55:07.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ac80aef50 0x7f0ac80b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient(hunting): picked mon.noname-a con 0x7f0ac80aef50 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:07.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac80b0ec0 0x7f0ac813fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient(hunting): picked mon.noname-b con 0x7f0ac80b0ec0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:07.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ac813cd30 0x7f0ac81364e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient(hunting): picked mon.noname-c con 0x7f0ac813cd30 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:07.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac8140880 con 0x7f0ac80aef50 2023-10-30T17:55:07.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.871 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac8141da0 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.871 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:07.871 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac813f9f0 con 0x7f0ac813cd30 2023-10-30T17:55:07.871 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:07.871 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.848+0000 7f0ace5b0640 10 monclient(hunting): authenticate will time out at 2690.171667s 2023-10-30T17:55:07.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ac80aef50 0x7f0ac80b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac80b0ec0 0x7f0ac813fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0accb4d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ac813cd30 0x7f0ac81364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac80b0ec0 0x7f0ac813fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:49600/0 (socket says 172.21.15.170:49600) 2023-10-30T17:55:07.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac77fe640 1 -- 172.21.15.170:0/2995897547 learned_addr learned my addr 172.21.15.170:0/2995897547 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:07.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac77fe640 10 monclient(hunting): get_auth_request con 0x7f0ac80b0ec0 auth_method 0 2023-10-30T17:55:07.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac7fff640 10 monclient(hunting): get_auth_request con 0x7f0ac80aef50 auth_method 0 2023-10-30T17:55:07.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0accb4d640 10 monclient(hunting): get_auth_request con 0x7f0ac813cd30 auth_method 0 2023-10-30T17:55:07.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0accb4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:07.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0accb4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:07.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0accb4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:07.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.849+0000 7f0ac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.850+0000 7f0accb4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:07.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.850+0000 7f0accb4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:07.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.850+0000 7f0accb4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:07.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.852+0000 7f0ac77fe640 10 monclient(hunting): handle_auth_done global_id 21493 payload 824 2023-10-30T17:55:07.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.852+0000 7f0ac77fe640 1 -- 172.21.15.170:0/2995897547 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ac813cd30 msgr2=0x7f0ac81364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.852+0000 7f0ac77fe640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ac813cd30 0x7f0ac81364e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.852+0000 7f0ac77fe640 1 -- 172.21.15.170:0/2995897547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ac80aef50 msgr2=0x7f0ac80b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:07.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.852+0000 7f0ac77fe640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ac80aef50 0x7f0ac80b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:07.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.852+0000 7f0ac77fe640 10 monclient: _finish_hunting 0 2023-10-30T17:55:07.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.852+0000 7f0ac77fe640 1 monclient: found mon.noname-b 2023-10-30T17:55:07.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.852+0000 7f0ac77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:07.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.852+0000 7f0ac77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:07.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.852+0000 7f0ac77fe640 1 -- 172.21.15.170:0/2995897547 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc009400 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.854+0000 7f0ac77fe640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac80b0ec0 0x7f0ac813fe50 secure :-1 s=READY pgs=5282 cs=0 l=1 rev1=1 crypto rx=0x7f0abc034040 tx=0x7f0abc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:07.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.854+0000 7f0ac4ff9640 10 client.?.objecter ms_handle_connect 0x7f0ac80b0ec0 2023-10-30T17:55:07.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.854+0000 7f0ac4ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:07.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ac4ff9640 1 -- 172.21.15.170:0/2995897547 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc032810 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ac4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:07.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ac4ff9640 10 monclient: dump: 2023-10-30T17:55:07.882 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.882 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.882 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.882 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.882 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.883 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.883 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.883 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.883 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.883 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ac4ff9640 10 monclient: _finish_auth 0 2023-10-30T17:55:07.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ac4ff9640 10 monclient: _check_auth_tickets 2023-10-30T17:55:07.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ac4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:07.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ac4ff9640 1 -- 172.21.15.170:0/2995897547 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc032990 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ace5b0640 5 monclient: authenticate success, global_id 21493 2023-10-30T17:55:07.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ace5b0640 10 monclient: _renew_subs 2023-10-30T17:55:07.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ace5b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:07.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.857+0000 7f0ace5b0640 1 -- 172.21.15.170:0/2995897547 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac8140390 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ac4ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:07.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ac4ff9640 1 -- 172.21.15.170:0/2995897547 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc032d40 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:07.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ac4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:07.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ac4ff9640 10 monclient: dump: 2023-10-30T17:55:07.886 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:07.886 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:07.886 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.887 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:07.887 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:07.887 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:07.887 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:07.887 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:07.887 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:07.888 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:07.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ac67fc640 4 set_mon_vals no callback set 2023-10-30T17:55:07.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ace5b0640 10 client.21493.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:07.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ace5b0640 10 monclient: _renew_subs 2023-10-30T17:55:07.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ace5b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:07.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ace5b0640 1 -- 172.21.15.170:0/2995897547 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac8136c00 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ace5b0640 1 librados: init done 2023-10-30T17:55:07.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ace5b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:07.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ace5b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:07.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ace5b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:07.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.858+0000 7f0ace5b0640 1 -- 172.21.15.170:0/2995897547 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac81406c0 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.876+0000 7f0ac4ff9640 1 -- 172.21.15.170:0/2995897547 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0abc04e440 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.876+0000 7f0ac4ff9640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f0ab406c850 0x7f0ab406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:07.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.877+0000 7f0ac7fff640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f0ab406c850 0x7f0ab406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:07.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.877+0000 7f0ac7fff640 10 monclient: get_auth_request con 0x7f0ab406c850 auth_method 0 2023-10-30T17:55:07.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.877+0000 7f0ac4ff9640 10 client.21493.objecter ms_handle_connect 0x7f0ab406c850 2023-10-30T17:55:07.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.877+0000 7f0ac7fff640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f0ab406c850 0x7f0ab406ed10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0ab80011e0 tx=0x7f0ab8007d30 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:07.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.878+0000 7f0ac4ff9640 1 -- 172.21.15.170:0/2995897547 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f0abc07a490 con 0x7f0ac80b0ec0 2023-10-30T17:55:07.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.878+0000 7f0ac4ff9640 10 client.21493.objecter ms_dispatch 0x7f0ac8000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:07.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.878+0000 7f0ac4ff9640 3 client.21493.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:07.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.878+0000 7f0ac4ff9640 3 client.21493.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:07.892 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:07.878+0000 7f0ac4ff9640 20 client.21493.objecter dump_active .. 0 homeless 2023-10-30T17:55:08.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.108+0000 7f0ac4ff9640 1 -- 172.21.15.170:0/2995897547 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0abc0767d0 con 0x7f0ac80b0ec0 2023-10-30T17:55:08.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.108+0000 7f0ac4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:08.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.108+0000 7f0ac4ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:08.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.283+0000 7f0ace5b0640 1 -- 172.21.15.170:0/2995897547 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ac80b1a70 con 0x7f0ab406c850 2023-10-30T17:55:08.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.284+0000 7f0ac4ff9640 1 -- 172.21.15.170:0/2995897547 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0ab8005850 con 0x7f0ab406c850 2023-10-30T17:55:08.285 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T17:55:08.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.285+0000 7f0ace5b0640 10 librados: watch_flush enter 2023-10-30T17:55:08.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.285+0000 7f0ace5b0640 10 librados: watch_flush exit 2023-10-30T17:55:08.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.285+0000 7f0ace5b0640 20 client.21493.objecter shutdown clearing up homeless session... 2023-10-30T17:55:08.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.285+0000 7f0ace5b0640 10 client.21493.objecter successfully canceled tick 2023-10-30T17:55:08.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.285+0000 7f0ace5b0640 1 -- 172.21.15.170:0/2995897547 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f0ab406c850 msgr2=0x7f0ab406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.285+0000 7f0ace5b0640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f0ab406c850 0x7f0ab406ed10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0ab80011e0 tx=0x7f0ab8007d30 comp rx=0 tx=0).stop 2023-10-30T17:55:08.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.285+0000 7f0ace5b0640 10 monclient: shutdown 2023-10-30T17:55:08.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.285+0000 7f0ace5b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:08.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.285+0000 7f0ace5b0640 1 -- 172.21.15.170:0/2995897547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac80b0ec0 msgr2=0x7f0ac813fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.285+0000 7f0ace5b0640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac80b0ec0 0x7f0ac813fe50 secure :-1 s=READY pgs=5282 cs=0 l=1 rev1=1 crypto rx=0x7f0abc034040 tx=0x7f0abc0046c0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.286+0000 7f0ace5b0640 1 -- 172.21.15.170:0/2995897547 shutdown_connections 2023-10-30T17:55:08.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.286+0000 7f0ace5b0640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f0ab406c850 0x7f0ab406ed10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.286+0000 7f0ace5b0640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac80b0ec0 0x7f0ac813fe50 unknown :-1 s=CLOSED pgs=5282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.286+0000 7f0ace5b0640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ac813cd30 0x7f0ac81364e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.286+0000 7f0ace5b0640 1 --2- 172.21.15.170:0/2995897547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ac80aef50 0x7f0ac80b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.286+0000 7f0ace5b0640 1 -- 172.21.15.170:0/2995897547 >> 172.21.15.170:0/2995897547 conn(0x7f0ac80b1d40 msgr2=0x7f0ac80b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:08.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.286+0000 7f0ace5b0640 1 -- 172.21.15.170:0/2995897547 shutdown_connections 2023-10-30T17:55:08.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.286+0000 7f0ace5b0640 1 -- 172.21.15.170:0/2995897547 wait complete. 2023-10-30T17:55:08.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.286+0000 7f0ace5b0640 1 librados: shutdown 2023-10-30T17:55:08.298 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T17:55:08.298 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T17:55:08.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.402+0000 7f7ac0283640 10 monclient: get_monmap_and_config 2023-10-30T17:55:08.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.402+0000 7f7ac0283640 10 monclient: build_initial_monmap 2023-10-30T17:55:08.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.402+0000 7f7ac0283640 10 monclient: monmap: 2023-10-30T17:55:08.408 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:08.408 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:08.408 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:08.403031+0000 2023-10-30T17:55:08.408 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:08.403031+0000 2023-10-30T17:55:08.408 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:08.409 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.409 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:08.409 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:08.409 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:08.409 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:08.410 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:08.410 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.410 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.402+0000 7f7ac0283640 1 Processor -- start 2023-10-30T17:55:08.410 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.402+0000 7f7ac0283640 1 -- start start 2023-10-30T17:55:08.410 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.402+0000 7f7ac0283640 10 monclient: init 2023-10-30T17:55:08.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:08.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T17:55:08.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ab80b4460 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:08.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ab813eb10 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:08.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 0x7f7ab80b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.412 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 10 monclient(hunting): picked mon.noname-b con 0x7f7ab80b08c0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:08.412 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab80b04a0 con 0x7f7ab80b4460 2023-10-30T17:55:08.412 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab80af010 con 0x7f7ab813eb10 2023-10-30T17:55:08.412 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:08.412 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab80b4840 con 0x7f7ab80b08c0 2023-10-30T17:55:08.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:08.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.403+0000 7f7ac0283640 10 monclient(hunting): authenticate will time out at 2690.726620s 2023-10-30T17:55:08.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.404+0000 7f7abe01f640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f7ab80b4460 0x7f7ab80ae860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59474/0 (socket says 172.21.15.170:59474) 2023-10-30T17:55:08.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.404+0000 7f7abe01f640 1 -- 172.21.15.170:0/750504312 learned_addr learned my addr 172.21.15.170:0/750504312 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:08.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.404+0000 7f7abe820640 1 --2- 172.21.15.170:0/750504312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 0x7f7ab80b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.404+0000 7f7abe820640 10 monclient(hunting): get_auth_request con 0x7f7ab80b08c0 auth_method 0 2023-10-30T17:55:08.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.404+0000 7f7abe820640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:08.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.404+0000 7f7abe820640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.404+0000 7f7abe820640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.405+0000 7f7abe820640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:08.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.405+0000 7f7abe820640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:08.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.405+0000 7f7abe820640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:08.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 1 -- 172.21.15.170:0/750504312 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512519568 0 0) 0x7f7ab80af010 con 0x7f7ab813eb10 2023-10-30T17:55:08.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 10 monclient(hunting): _negotiate 2023-10-30T17:55:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 10 monclient(hunting): my global_id is 21496 2023-10-30T17:55:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 1 -- 172.21.15.170:0/750504312 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa4003520 con 0x7f7ab813eb10 2023-10-30T17:55:08.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 1 -- 172.21.15.170:0/750504312 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419481052 0 0) 0x7f7ab80b04a0 con 0x7f7ab80b4460 2023-10-30T17:55:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 10 monclient(hunting): _negotiate 2023-10-30T17:55:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 10 monclient(hunting): my global_id is 16050 2023-10-30T17:55:08.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abd01d640 1 -- 172.21.15.170:0/750504312 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab80af010 con 0x7f7ab80b4460 2023-10-30T17:55:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abe820640 10 monclient(hunting): handle_auth_done global_id 16358 payload 274 2023-10-30T17:55:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abe820640 1 -- 172.21.15.170:0/750504312 >> v1:172.21.15.177:6789/0 conn(0x7f7ab813eb10 legacy=0x7f7ab813ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abe820640 1 -- 172.21.15.170:0/750504312 >> v1:172.21.15.170:6789/0 conn(0x7f7ab80b4460 legacy=0x7f7ab80ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abe820640 10 monclient: _finish_hunting 0 2023-10-30T17:55:08.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abe820640 1 monclient: found mon.noname-b 2023-10-30T17:55:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abe820640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abe820640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:55:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.406+0000 7f7abe820640 1 -- 172.21.15.170:0/750504312 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab80b3a40 con 0x7f7ab80b08c0 2023-10-30T17:55:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.408+0000 7f7abe820640 1 --2- 172.21.15.170:0/750504312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 0x7f7ab80b0d50 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00097d0 tx=0x7f7ab002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72e9b1e5cd97dc9a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:08.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 1 -- 172.21.15.170:0/750504312 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0031800 con 0x7f7ab80b08c0 2023-10-30T17:55:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 10 monclient: dump: 2023-10-30T17:55:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:08.424 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:08.425 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:08.426 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 10 monclient: _finish_auth 0 2023-10-30T17:55:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 10 monclient: _check_auth_tickets 2023-10-30T17:55:08.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 1 -- 172.21.15.170:0/750504312 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab0031960 con 0x7f7ab80b08c0 2023-10-30T17:55:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7ac0283640 5 monclient: authenticate success, global_id 16358 2023-10-30T17:55:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7ac0283640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:08.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7ac0283640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 1 -- 172.21.15.170:0/750504312 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0031cf0 con 0x7f7ab80b08c0 2023-10-30T17:55:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7ac0283640 4 set_mon_vals no callback set 2023-10-30T17:55:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:55:08.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.413+0000 7f7abd01d640 10 monclient: dump: 2023-10-30T17:55:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.429 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:08.430 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:08.430 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.414+0000 7f7ac0283640 10 monclient: shutdown 2023-10-30T17:55:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.414+0000 7f7ac0283640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.414+0000 7f7ac0283640 1 -- 172.21.15.170:0/750504312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 msgr2=0x7f7ab80b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.414+0000 7f7ac0283640 1 --2- 172.21.15.170:0/750504312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 0x7f7ab80b0d50 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00097d0 tx=0x7f7ab002b8a0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.414+0000 7f7ac0283640 1 -- 172.21.15.170:0/750504312 shutdown_connections 2023-10-30T17:55:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.414+0000 7f7ac0283640 1 --2- 172.21.15.170:0/750504312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 0x7f7ab80b0d50 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.414+0000 7f7ac0283640 1 -- 172.21.15.170:0/750504312 >> 172.21.15.170:0/750504312 conn(0x7f7ab8071b90 msgr2=0x7f7ab8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:08.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.414+0000 7f7ac0283640 1 -- 172.21.15.170:0/750504312 shutdown_connections 2023-10-30T17:55:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.414+0000 7f7ac0283640 1 -- 172.21.15.170:0/750504312 wait complete. 2023-10-30T17:55:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.415+0000 7f7ac0283640 10 monclient: build_initial_monmap 2023-10-30T17:55:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.415+0000 7f7ac0283640 10 monclient: monmap: 2023-10-30T17:55:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:08.433 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:08.434 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:08.434 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:08.434 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:08.434 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.434 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:08.435 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:08.435 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:08.435 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.435 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.415+0000 7f7ac0283640 1 librados: starting msgr at 2023-10-30T17:55:08.435 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.415+0000 7f7ac0283640 1 librados: starting objecter 2023-10-30T17:55:08.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.415+0000 7f7ac0283640 1 Processor -- start 2023-10-30T17:55:08.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.415+0000 7f7ac0283640 1 -- start start 2023-10-30T17:55:08.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.415+0000 7f7ac0283640 1 librados: setting wanted keys 2023-10-30T17:55:08.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.415+0000 7f7ac0283640 1 librados: calling monclient init 2023-10-30T17:55:08.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.415+0000 7f7ac0283640 10 monclient: init 2023-10-30T17:55:08.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:08.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T17:55:08.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 0x7f7ab8134f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 10 monclient(hunting): picked mon.noname-c con 0x7f7ab80b08c0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:08.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ab80b4460 0x7f7ab8135cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 10 monclient(hunting): picked mon.noname-a con 0x7f7ab80b4460 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:08.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ab813eb10 0x7f7ab81d6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 10 monclient(hunting): picked mon.noname-b con 0x7f7ab813eb10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:08.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:08.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab8137530 con 0x7f7ab80b4460 2023-10-30T17:55:08.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:08.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab8137950 con 0x7f7ab813eb10 2023-10-30T17:55:08.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:08.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab8137bf0 con 0x7f7ab80b08c0 2023-10-30T17:55:08.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:08.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7ac0283640 10 monclient(hunting): authenticate will time out at 2690.739200s 2023-10-30T17:55:08.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abe01f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 0x7f7ab8134f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abe820640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ab813eb10 0x7f7ab81d6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abd81e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ab80b4460 0x7f7ab8135cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abe820640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ab813eb10 0x7f7ab81d6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:49614/0 (socket says 172.21.15.170:49614) 2023-10-30T17:55:08.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abe01f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 0x7f7ab8134f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:39104/0 (socket says 172.21.15.170:39104) 2023-10-30T17:55:08.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abe820640 1 -- 172.21.15.170:0/3808685299 learned_addr learned my addr 172.21.15.170:0/3808685299 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:08.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abd81e640 10 monclient(hunting): get_auth_request con 0x7f7ab80b4460 auth_method 0 2023-10-30T17:55:08.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abd81e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:08.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abd81e640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abd81e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abe820640 10 monclient(hunting): get_auth_request con 0x7f7ab813eb10 auth_method 0 2023-10-30T17:55:08.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abe820640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:08.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abe820640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.416+0000 7f7abe820640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abe01f640 10 monclient(hunting): get_auth_request con 0x7f7ab80b08c0 auth_method 0 2023-10-30T17:55:08.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abe01f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:08.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abe01f640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abe01f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abe820640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:08.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abe820640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:08.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abe820640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:08.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abd81e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:08.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abd81e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:08.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abd81e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:08.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abe01f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:08.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abe01f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:08.445 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.417+0000 7f7abe01f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:08.446 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.420+0000 7f7abe820640 10 monclient(hunting): handle_auth_done global_id 21499 payload 824 2023-10-30T17:55:08.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.420+0000 7f7abe820640 1 -- 172.21.15.170:0/3808685299 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 msgr2=0x7f7ab8134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.420+0000 7f7abe820640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ab80b08c0 0x7f7ab8134f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.420+0000 7f7abe820640 1 -- 172.21.15.170:0/3808685299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ab80b4460 msgr2=0x7f7ab8135cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.420+0000 7f7abe820640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ab80b4460 0x7f7ab8135cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.420+0000 7f7abe820640 10 monclient: _finish_hunting 0 2023-10-30T17:55:08.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.420+0000 7f7abe820640 1 monclient: found mon.noname-b 2023-10-30T17:55:08.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.420+0000 7f7abe820640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:08.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.420+0000 7f7abe820640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:08.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.420+0000 7f7abe820640 1 -- 172.21.15.170:0/3808685299 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab0009400 con 0x7f7ab813eb10 2023-10-30T17:55:08.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.422+0000 7f7abe820640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ab813eb10 0x7f7ab81d6420 secure :-1 s=READY pgs=5284 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00098f0 tx=0x7f7ab00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:08.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.422+0000 7f7aaeffd640 10 client.?.objecter ms_handle_connect 0x7f7ab813eb10 2023-10-30T17:55:08.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.422+0000 7f7aaeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:08.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7aaeffd640 1 -- 172.21.15.170:0/3808685299 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab003bbf0 con 0x7f7ab813eb10 2023-10-30T17:55:08.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7aaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:08.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7aaeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:08.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7aaeffd640 10 monclient: dump: 2023-10-30T17:55:08.450 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:08.450 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:08.450 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.450 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.451 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:08.451 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.451 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:08.451 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:08.451 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:08.452 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7aaeffd640 10 monclient: _finish_auth 0 2023-10-30T17:55:08.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7aaeffd640 10 monclient: _check_auth_tickets 2023-10-30T17:55:08.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7aaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:08.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7ac0283640 5 monclient: authenticate success, global_id 21499 2023-10-30T17:55:08.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7ac0283640 10 monclient: _renew_subs 2023-10-30T17:55:08.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7ac0283640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:08.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab8136210 con 0x7f7ab813eb10 2023-10-30T17:55:08.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7ac0283640 10 client.21499.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:08.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7ac0283640 10 monclient: _renew_subs 2023-10-30T17:55:08.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7ac0283640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:08.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab81d6b40 con 0x7f7ab813eb10 2023-10-30T17:55:08.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.424+0000 7f7ac0283640 1 librados: init done 2023-10-30T17:55:08.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.425+0000 7f7ac0283640 10 client.21499.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:08.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.425+0000 7f7ac0283640 15 client.21499.objecter _session_command_op_assign -1 1 2023-10-30T17:55:08.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.425+0000 7f7ac0283640 20 client.21499.objecter _get_session osd=-1 returning homeless 2023-10-30T17:55:08.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.425+0000 7f7ac0283640 10 client.21499.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:08.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.425+0000 7f7ac0283640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:08.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.425+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7ab8137e10 con 0x7f7ab813eb10 2023-10-30T17:55:08.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.426+0000 7f7aaeffd640 1 -- 172.21.15.170:0/3808685299 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab8137e10 con 0x7f7ab813eb10 2023-10-30T17:55:08.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.426+0000 7f7aaeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:08.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.426+0000 7f7aaeffd640 1 -- 172.21.15.170:0/3808685299 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0006e10 con 0x7f7ab813eb10 2023-10-30T17:55:08.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.426+0000 7f7aaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:08.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.426+0000 7f7aaeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:08.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.426+0000 7f7aaeffd640 10 monclient: dump: 2023-10-30T17:55:08.456 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:08.457 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:08.457 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.457 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.457 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:08.457 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.458 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:08.458 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:08.458 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:08.458 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.426+0000 7f7abc81c640 4 set_mon_vals no callback set 2023-10-30T17:55:08.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.446+0000 7f7aaeffd640 1 -- 172.21.15.170:0/3808685299 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7ab004e3f0 con 0x7f7ab813eb10 2023-10-30T17:55:08.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.446+0000 7f7aaeffd640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f7a9c06c800 0x7f7a9c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.447+0000 7f7abe01f640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f7a9c06c800 0x7f7a9c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.447+0000 7f7abe01f640 10 monclient: get_auth_request con 0x7f7a9c06c800 auth_method 0 2023-10-30T17:55:08.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.447+0000 7f7abe01f640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f7a9c06c800 0x7f7a9c06ecc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4001560 tx=0x7f7ab4001a90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:08.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.447+0000 7f7aaeffd640 10 client.21499.objecter ms_handle_connect 0x7f7a9c06c800 2023-10-30T17:55:08.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 1 -- 172.21.15.170:0/3808685299 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f7ab002e070 con 0x7f7ab813eb10 2023-10-30T17:55:08.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 10 client.21499.objecter ms_dispatch 0x7f7ab8000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:08.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 3 client.21499.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:08.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 3 client.21499.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:08.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 10 client.21499.objecter checking command 1 2023-10-30T17:55:08.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] conn(0x7f7a9c074010 0x7f7a9c0764b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 20 client.21499.objecter _get_session s=0x7f7a9c073940 osd=5 3 2023-10-30T17:55:08.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 20 client.21499.objecter put_session s=0x7f7a9c073940 osd=5 3 2023-10-30T17:55:08.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 15 client.21499.objecter _session_command_op_remove -1 1 2023-10-30T17:55:08.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 20 client.21499.objecter _get_session s=0x7f7a9c073940 osd=5 3 2023-10-30T17:55:08.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 20 client.21499.objecter get_session s=0x7f7a9c073940 osd=5 3 2023-10-30T17:55:08.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 15 client.21499.objecter _session_command_op_assign 5 1 2023-10-30T17:55:08.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 20 client.21499.objecter put_session s=0x7f7a9c073940 osd=5 4 2023-10-30T17:55:08.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 10 client.21499.objecter _send_command 1 2023-10-30T17:55:08.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 1 -- 172.21.15.170:0/3808685299 --> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a9c076b80 con 0x7f7a9c074010 2023-10-30T17:55:08.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 20 client.21499.objecter dump_active .. 0 homeless 2023-10-30T17:55:08.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 1 -- 172.21.15.170:0/3808685299 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=914) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ab0077340 con 0x7f7ab813eb10 2023-10-30T17:55:08.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7aaeffd640 10 monclient: handle_get_version_reply finishing 1 version 914 2023-10-30T17:55:08.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.449+0000 7f7abd81e640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] conn(0x7f7a9c074010 0x7f7a9c0764b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.450+0000 7f7abd81e640 10 monclient: get_auth_request con 0x7f7a9c074010 auth_method 0 2023-10-30T17:55:08.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.452+0000 7f7abd81e640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] conn(0x7f7a9c074010 0x7f7a9c0764b0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:08.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.452+0000 7f7aaeffd640 10 client.21499.objecter ms_handle_connect 0x7f7a9c074010 2023-10-30T17:55:08.476 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.475+0000 7f7aaeffd640 1 -- 172.21.15.170:0/3808685299 <== osd.5 v2:172.21.15.177:6808/2243649533 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f7a9c076b80 con 0x7f7a9c074010 2023-10-30T17:55:08.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.475+0000 7f7aaeffd640 10 client.21499.objecter ms_dispatch 0x7f7ab8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:55:08.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.475+0000 7f7aaeffd640 10 client.21499.objecter _finish_command 1 = system:0 2023-10-30T17:55:08.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.475+0000 7f7aaeffd640 20 client.21499.objecter put_session s=0x7f7a9c073940 osd=5 4 2023-10-30T17:55:08.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.475+0000 7f7aaeffd640 15 client.21499.objecter _session_command_op_remove 5 1 2023-10-30T17:55:08.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 10 client.21499.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T17:55:08.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 15 client.21499.objecter _session_command_op_assign -1 2 2023-10-30T17:55:08.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 20 client.21499.objecter _get_session s=0x7f7a9c073940 osd=5 3 2023-10-30T17:55:08.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 20 client.21499.objecter put_session s=0x7f7a9c073940 osd=5 3 2023-10-30T17:55:08.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 20 client.21499.objecter _get_session s=0x7f7a9c073940 osd=5 3 2023-10-30T17:55:08.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 15 client.21499.objecter _session_command_op_remove -1 2 2023-10-30T17:55:08.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 20 client.21499.objecter get_session s=0x7f7a9c073940 osd=5 3 2023-10-30T17:55:08.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 15 client.21499.objecter _session_command_op_assign 5 2 2023-10-30T17:55:08.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 20 client.21499.objecter put_session s=0x7f7a9c073940 osd=5 4 2023-10-30T17:55:08.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 10 client.21499.objecter _send_command 2 2023-10-30T17:55:08.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.487+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 --> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7ab8067cd0 con 0x7f7a9c074010 2023-10-30T17:55:08.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7aaeffd640 1 -- 172.21.15.170:0/3808685299 <== osd.5 v2:172.21.15.177:6808/2243649533 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f7ab8067cd0 con 0x7f7a9c074010 2023-10-30T17:55:08.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7aaeffd640 10 client.21499.objecter ms_dispatch 0x7f7ab8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:55:08.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7aaeffd640 10 client.21499.objecter _finish_command 2 = system:0 2023-10-30T17:55:08.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7aaeffd640 20 client.21499.objecter put_session s=0x7f7a9c073940 osd=5 4 2023-10-30T17:55:08.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7aaeffd640 15 client.21499.objecter _session_command_op_remove 5 2 2023-10-30T17:55:08.504 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-30T17:55:08.504 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T17:55:08.504 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-30T17:55:08.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7ac0283640 10 librados: watch_flush enter 2023-10-30T17:55:08.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7ac0283640 10 librados: watch_flush exit 2023-10-30T17:55:08.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7ac0283640 10 client.21499.objecter close_session for osd.5 2023-10-30T17:55:08.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] conn(0x7f7a9c074010 msgr2=0x7f7a9c0764b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7ac0283640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:6808/2243649533,v1:172.21.15.177:6810/2243649533] conn(0x7f7a9c074010 0x7f7a9c0764b0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7ac0283640 20 client.21499.objecter put_session s=0x7f7a9c073940 osd=5 1 2023-10-30T17:55:08.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7ac0283640 20 client.21499.objecter shutdown clearing up homeless session... 2023-10-30T17:55:08.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.501+0000 7f7ac0283640 10 client.21499.objecter successfully canceled tick 2023-10-30T17:55:08.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f7a9c06c800 msgr2=0x7f7a9c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f7a9c06c800 0x7f7a9c06ecc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4001560 tx=0x7f7ab4001a90 comp rx=0 tx=0).stop 2023-10-30T17:55:08.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 10 monclient: shutdown 2023-10-30T17:55:08.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:08.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ab813eb10 msgr2=0x7f7ab81d6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 1 --2- 172.21.15.170:0/3808685299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ab813eb10 0x7f7ab81d6420 secure :-1 s=READY pgs=5284 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00098f0 tx=0x7f7ab00046c0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7abe820640 1 -- 172.21.15.170:0/3808685299 reap_dead start 2023-10-30T17:55:08.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 shutdown_connections 2023-10-30T17:55:08.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 >> 172.21.15.170:0/3808685299 conn(0x7f7ab8071b90 msgr2=0x7f7ab80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:08.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 shutdown_connections 2023-10-30T17:55:08.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 1 -- 172.21.15.170:0/3808685299 wait complete. 2023-10-30T17:55:08.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.502+0000 7f7ac0283640 1 librados: shutdown 2023-10-30T17:55:08.513 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T17:55:08.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.616+0000 7f8bd73bb640 10 monclient: get_monmap_and_config 2023-10-30T17:55:08.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.616+0000 7f8bd73bb640 10 monclient: build_initial_monmap 2023-10-30T17:55:08.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.616+0000 7f8bd73bb640 10 monclient: monmap: 2023-10-30T17:55:08.618 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:08.618 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:08.618 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:08.616982+0000 2023-10-30T17:55:08.618 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:08.616982+0000 2023-10-30T17:55:08.618 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:08.619 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.619 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:08.619 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:08.619 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:08.619 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:08.620 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:08.620 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.616+0000 7f8bd73bb640 1 Processor -- start 2023-10-30T17:55:08.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.616+0000 7f8bd73bb640 1 -- start start 2023-10-30T17:55:08.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.616+0000 7f8bd73bb640 10 monclient: init 2023-10-30T17:55:08.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:08.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-10-30T17:55:08.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8bd00b0330 0x7f8bd00a5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 10 monclient(hunting): picked mon.noname-c con 0x7f8bd00b0330 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:08.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bd00a8e90 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:08.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bd00ab960 0x7f8bd00abdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 10 monclient(hunting): picked mon.noname-b con 0x7f8bd00ab960 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:08.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd00aca70 con 0x7f8bd00a8e90 2023-10-30T17:55:08.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd00ac7d0 con 0x7f8bd00b0330 2023-10-30T17:55:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd00b0710 con 0x7f8bd00ab960 2023-10-30T17:55:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd73bb640 10 monclient(hunting): authenticate will time out at 2690.940494s 2023-10-30T17:55:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd4956640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f8bd00a8e90 0x7f8bd00a92f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59480/0 (socket says 172.21.15.170:59480) 2023-10-30T17:55:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.617+0000 7f8bd4956640 1 -- 172.21.15.170:0/3979424819 learned_addr learned my addr 172.21.15.170:0/3979424819 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5958640 1 --2- 172.21.15.170:0/3979424819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bd00ab960 0x7f8bd00abdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5157640 1 --2- 172.21.15.170:0/3979424819 >> v2:172.21.15.177:3300/0 conn(0x7f8bd00b0330 0x7f8bd00a5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5157640 10 monclient(hunting): get_auth_request con 0x7f8bd00b0330 auth_method 0 2023-10-30T17:55:08.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5157640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:08.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5157640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5157640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5958640 10 monclient(hunting): get_auth_request con 0x7f8bd00ab960 auth_method 0 2023-10-30T17:55:08.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5958640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5958640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5958640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5157640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5157640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.618+0000 7f8bd5157640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:08.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.619+0000 7f8bd5958640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:08.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.619+0000 7f8bd5958640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:08.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.619+0000 7f8bd5958640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:08.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bc7fff640 1 -- 172.21.15.170:0/3979424819 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013912317 0 0) 0x7f8bd00aca70 con 0x7f8bd00a8e90 2023-10-30T17:55:08.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bc7fff640 10 monclient(hunting): _negotiate 2023-10-30T17:55:08.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bc7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bc7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bc7fff640 10 monclient(hunting): my global_id is 16056 2023-10-30T17:55:08.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bc7fff640 1 -- 172.21.15.170:0/3979424819 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bbc005730 con 0x7f8bd00a8e90 2023-10-30T17:55:08.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bd5157640 10 monclient(hunting): handle_auth_done global_id 21502 payload 274 2023-10-30T17:55:08.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bd5157640 1 -- 172.21.15.170:0/3979424819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bd00ab960 msgr2=0x7f8bd00abdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:08.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bd5157640 1 --2- 172.21.15.170:0/3979424819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bd00ab960 0x7f8bd00abdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bd5157640 1 -- 172.21.15.170:0/3979424819 >> v1:172.21.15.170:6789/0 conn(0x7f8bd00a8e90 legacy=0x7f8bd00a92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bd5157640 10 monclient: _finish_hunting 0 2023-10-30T17:55:08.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bd5157640 1 monclient: found mon.noname-c 2023-10-30T17:55:08.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bd5157640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:08.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bd5157640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:55:08.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.620+0000 7f8bd5157640 1 -- 172.21.15.170:0/3979424819 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd00a73c0 con 0x7f8bd00b0330 2023-10-30T17:55:08.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.622+0000 7f8bd5157640 1 --2- 172.21.15.170:0/3979424819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00b0330 0x7f8bd00a5c90 secure :-1 s=READY pgs=5285 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80097a0 tx=0x7f8bc802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1697c5b03208cf59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:08.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.624+0000 7f8bc7fff640 1 -- 172.21.15.170:0/3979424819 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8031830 con 0x7f8bd00b0330 2023-10-30T17:55:08.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.624+0000 7f8bc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:08.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.624+0000 7f8bc7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:08.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.624+0000 7f8bc7fff640 10 monclient: dump: 2023-10-30T17:55:08.637 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:08.637 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:08.637 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.637 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.638 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:08.638 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.638 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:08.638 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:08.638 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:08.639 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.624+0000 7f8bc7fff640 10 monclient: _finish_auth 0 2023-10-30T17:55:08.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.624+0000 7f8bc7fff640 10 monclient: _check_auth_tickets 2023-10-30T17:55:08.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.624+0000 7f8bc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:08.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.624+0000 7f8bd73bb640 5 monclient: authenticate success, global_id 21502 2023-10-30T17:55:08.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.624+0000 7f8bd73bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:08.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bc7fff640 1 -- 172.21.15.170:0/3979424819 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc8031990 con 0x7f8bd00b0330 2023-10-30T17:55:08.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bc7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:08.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bc7fff640 1 -- 172.21.15.170:0/3979424819 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8031d00 con 0x7f8bd00b0330 2023-10-30T17:55:08.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:08.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 4 set_mon_vals no callback set 2023-10-30T17:55:08.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:08.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bc7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:08.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bc7fff640 10 monclient: dump: 2023-10-30T17:55:08.641 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:08.641 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:08.642 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.642 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.642 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:08.642 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.642 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:08.643 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:08.643 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:08.643 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 10 monclient: shutdown 2023-10-30T17:55:08.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:08.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 1 -- 172.21.15.170:0/3979424819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00b0330 msgr2=0x7f8bd00a5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 1 --2- 172.21.15.170:0/3979424819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00b0330 0x7f8bd00a5c90 secure :-1 s=READY pgs=5285 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80097a0 tx=0x7f8bc802b8d0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 1 -- 172.21.15.170:0/3979424819 shutdown_connections 2023-10-30T17:55:08.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 1 --2- 172.21.15.170:0/3979424819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bd00ab960 0x7f8bd00abdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 1 --2- 172.21.15.170:0/3979424819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00b0330 0x7f8bd00a5c90 unknown :-1 s=CLOSED pgs=5285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 1 -- 172.21.15.170:0/3979424819 >> 172.21.15.170:0/3979424819 conn(0x7f8bd00b3560 msgr2=0x7f8bd00af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:08.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 1 -- 172.21.15.170:0/3979424819 shutdown_connections 2023-10-30T17:55:08.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.626+0000 7f8bd73bb640 1 -- 172.21.15.170:0/3979424819 wait complete. 2023-10-30T17:55:08.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.627+0000 7f8bd73bb640 10 monclient: build_initial_monmap 2023-10-30T17:55:08.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.627+0000 7f8bd73bb640 10 monclient: monmap: 2023-10-30T17:55:08.646 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:08.646 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:08.646 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:08.646 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:08.647 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:08.647 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.647 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:08.647 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:08.647 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:08.647 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.627+0000 7f8bd73bb640 1 librados: starting msgr at 2023-10-30T17:55:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.627+0000 7f8bd73bb640 1 librados: starting objecter 2023-10-30T17:55:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.627+0000 7f8bd73bb640 1 Processor -- start 2023-10-30T17:55:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 1 -- start start 2023-10-30T17:55:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 1 librados: setting wanted keys 2023-10-30T17:55:08.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 1 librados: calling monclient init 2023-10-30T17:55:08.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient: init 2023-10-30T17:55:08.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:08.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:55:08.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00a8e90 0x7f8bd01c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.650 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient(hunting): picked mon.noname-b con 0x7f8bd00a8e90 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:08.650 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bd00ab960 0x7f8bd00704f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.650 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient(hunting): picked mon.noname-a con 0x7f8bd00ab960 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:08.650 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bd00b0330 0x7f8bd01d5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.650 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient(hunting): picked mon.noname-c con 0x7f8bd00b0330 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:08.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:08.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd0070a30 con 0x7f8bd00ab960 2023-10-30T17:55:08.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:08.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd0070060 con 0x7f8bd00a8e90 2023-10-30T17:55:08.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:08.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd006fd60 con 0x7f8bd00b0330 2023-10-30T17:55:08.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:08.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.628+0000 7f8bd73bb640 10 monclient(hunting): authenticate will time out at 2690.951666s 2023-10-30T17:55:08.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5157640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00a8e90 0x7f8bd01c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd4956640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bd00ab960 0x7f8bd00704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5157640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00a8e90 0x7f8bd01c5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:49634/0 (socket says 172.21.15.170:49634) 2023-10-30T17:55:08.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5157640 1 -- 172.21.15.170:0/1964614005 learned_addr learned my addr 172.21.15.170:0/1964614005 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:08.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5958640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bd00b0330 0x7f8bd01d5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5157640 10 monclient(hunting): get_auth_request con 0x7f8bd00a8e90 auth_method 0 2023-10-30T17:55:08.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5157640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:08.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5157640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5157640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd4956640 10 monclient(hunting): get_auth_request con 0x7f8bd00ab960 auth_method 0 2023-10-30T17:55:08.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd4956640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:08.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd4956640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd4956640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5958640 10 monclient(hunting): get_auth_request con 0x7f8bd00b0330 auth_method 0 2023-10-30T17:55:08.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5958640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:08.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5958640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:08.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5958640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:08.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5157640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:08.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5157640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:08.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd5157640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:08.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd4956640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:08.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd4956640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:08.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.629+0000 7f8bd4956640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:08.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.630+0000 7f8bd5958640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:08.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.630+0000 7f8bd5958640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:08.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.630+0000 7f8bd5958640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:08.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.632+0000 7f8bd5157640 10 monclient(hunting): handle_auth_done global_id 21505 payload 824 2023-10-30T17:55:08.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.632+0000 7f8bd5157640 1 -- 172.21.15.170:0/1964614005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bd00b0330 msgr2=0x7f8bd01d5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.632+0000 7f8bd5157640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bd00b0330 0x7f8bd01d5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.632+0000 7f8bd5157640 1 -- 172.21.15.170:0/1964614005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bd00ab960 msgr2=0x7f8bd00704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:08.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.632+0000 7f8bd5157640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bd00ab960 0x7f8bd00704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:08.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.632+0000 7f8bd5157640 10 monclient: _finish_hunting 0 2023-10-30T17:55:08.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.632+0000 7f8bd5157640 1 monclient: found mon.noname-b 2023-10-30T17:55:08.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.632+0000 7f8bd5157640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:08.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.632+0000 7f8bd5157640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:08.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.632+0000 7f8bd5157640 1 -- 172.21.15.170:0/1964614005 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc8009400 con 0x7f8bd00a8e90 2023-10-30T17:55:08.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.634+0000 7f8bd5157640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00a8e90 0x7f8bd01c5170 secure :-1 s=READY pgs=5286 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80098c0 tx=0x7f8bc80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:08.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.634+0000 7f8bc5ffb640 10 client.?.objecter ms_handle_connect 0x7f8bd00a8e90 2023-10-30T17:55:08.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.634+0000 7f8bc5ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:08.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 1 -- 172.21.15.170:0/1964614005 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8004440 con 0x7f8bd00a8e90 2023-10-30T17:55:08.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:08.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:08.661 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 10 monclient: dump: 2023-10-30T17:55:08.661 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:08.662 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:08.662 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.662 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.662 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:08.662 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.663 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:08.663 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:08.663 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:08.663 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 10 monclient: _finish_auth 0 2023-10-30T17:55:08.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:55:08.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:08.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 1 -- 172.21.15.170:0/1964614005 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc8003610 con 0x7f8bd00a8e90 2023-10-30T17:55:08.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bd73bb640 5 monclient: authenticate success, global_id 21505 2023-10-30T17:55:08.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bd73bb640 10 monclient: _renew_subs 2023-10-30T17:55:08.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bd73bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:08.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bd73bb640 1 -- 172.21.15.170:0/1964614005 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd01d63b0 con 0x7f8bd00a8e90 2023-10-30T17:55:08.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:08.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 1 -- 172.21.15.170:0/1964614005 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8003980 con 0x7f8bd00a8e90 2023-10-30T17:55:08.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:08.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:08.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc5ffb640 10 monclient: dump: 2023-10-30T17:55:08.666 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:08.666 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:08.666 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.667 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:08.667 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:08.667 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:08.667 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:08.667 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:08.667 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:08.668 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:08.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bc77fe640 4 set_mon_vals no callback set 2023-10-30T17:55:08.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bd73bb640 10 client.21505.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:08.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bd73bb640 10 monclient: _renew_subs 2023-10-30T17:55:08.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bd73bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:08.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bd73bb640 1 -- 172.21.15.170:0/1964614005 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd006f530 con 0x7f8bd00a8e90 2023-10-30T17:55:08.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.638+0000 7f8bd73bb640 1 librados: init done 2023-10-30T17:55:08.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.639+0000 7f8bd73bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:08.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.639+0000 7f8bd73bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:08.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.639+0000 7f8bd73bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:08.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.639+0000 7f8bd73bb640 1 -- 172.21.15.170:0/1964614005 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd00abdf0 con 0x7f8bd00a8e90 2023-10-30T17:55:08.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.660+0000 7f8bc5ffb640 1 -- 172.21.15.170:0/1964614005 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8bc8003ae0 con 0x7f8bd00a8e90 2023-10-30T17:55:08.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.660+0000 7f8bc5ffb640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8bb406c850 0x7f8bb406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:08.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.660+0000 7f8bd4956640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8bb406c850 0x7f8bb406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:08.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.660+0000 7f8bd4956640 10 monclient: get_auth_request con 0x7f8bb406c850 auth_method 0 2023-10-30T17:55:08.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.661+0000 7f8bc5ffb640 10 client.21505.objecter ms_handle_connect 0x7f8bb406c850 2023-10-30T17:55:08.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.661+0000 7f8bd4956640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8bb406c850 0x7f8bb406ed10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8bd01c8910 tx=0x7f8bc0007d30 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:08.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.662+0000 7f8bc5ffb640 1 -- 172.21.15.170:0/1964614005 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f8bc8041030 con 0x7f8bd00a8e90 2023-10-30T17:55:08.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.662+0000 7f8bc5ffb640 10 client.21505.objecter ms_dispatch 0x7f8bd0000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:08.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.662+0000 7f8bc5ffb640 3 client.21505.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:08.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.662+0000 7f8bc5ffb640 3 client.21505.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:08.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.662+0000 7f8bc5ffb640 20 client.21505.objecter dump_active .. 0 homeless 2023-10-30T17:55:08.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.892+0000 7f8bc5ffb640 1 -- 172.21.15.170:0/1964614005 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8bc8076df0 con 0x7f8bd00a8e90 2023-10-30T17:55:08.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.892+0000 7f8bc5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:08.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:08.892+0000 7f8bc5ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:09.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.061+0000 7f8bd73bb640 1 -- 172.21.15.170:0/1964614005 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bd013e810 con 0x7f8bb406c850 2023-10-30T17:55:09.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.062+0000 7f8bc5ffb640 1 -- 172.21.15.170:0/1964614005 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8bc0005850 con 0x7f8bb406c850 2023-10-30T17:55:09.064 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T17:55:09.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 10 librados: watch_flush enter 2023-10-30T17:55:09.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 10 librados: watch_flush exit 2023-10-30T17:55:09.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 20 client.21505.objecter shutdown clearing up homeless session... 2023-10-30T17:55:09.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 10 client.21505.objecter successfully canceled tick 2023-10-30T17:55:09.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 -- 172.21.15.170:0/1964614005 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8bb406c850 msgr2=0x7f8bb406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8bb406c850 0x7f8bb406ed10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8bd01c8910 tx=0x7f8bc0007d30 comp rx=0 tx=0).stop 2023-10-30T17:55:09.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 10 monclient: shutdown 2023-10-30T17:55:09.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:09.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 -- 172.21.15.170:0/1964614005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00a8e90 msgr2=0x7f8bd01c5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00a8e90 0x7f8bd01c5170 secure :-1 s=READY pgs=5286 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80098c0 tx=0x7f8bc80046c0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 -- 172.21.15.170:0/1964614005 shutdown_connections 2023-10-30T17:55:09.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f8bb406c850 0x7f8bb406ed10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bd00b0330 0x7f8bd01d5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bd00ab960 0x7f8bd00704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 --2- 172.21.15.170:0/1964614005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bd00a8e90 0x7f8bd01c5170 unknown :-1 s=CLOSED pgs=5286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 -- 172.21.15.170:0/1964614005 >> 172.21.15.170:0/1964614005 conn(0x7f8bd00b3560 msgr2=0x7f8bd00aacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:09.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 -- 172.21.15.170:0/1964614005 shutdown_connections 2023-10-30T17:55:09.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 -- 172.21.15.170:0/1964614005 wait complete. 2023-10-30T17:55:09.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.064+0000 7f8bd73bb640 1 librados: shutdown 2023-10-30T17:55:09.077 INFO:tasks.ceph:Scrubbing osd.6 2023-10-30T17:55:09.077 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-30T17:55:09.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.180+0000 7fe96e864640 10 monclient: get_monmap_and_config 2023-10-30T17:55:09.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.181+0000 7fe96e864640 10 monclient: build_initial_monmap 2023-10-30T17:55:09.182 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.181+0000 7fe96e864640 10 monclient: monmap: 2023-10-30T17:55:09.183 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:09.183 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:09.183 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:09.181954+0000 2023-10-30T17:55:09.183 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:09.181954+0000 2023-10-30T17:55:09.183 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:09.184 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.184 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:09.184 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:09.184 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:09.184 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:09.184 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:09.185 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.187 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.181+0000 7fe96e864640 1 Processor -- start 2023-10-30T17:55:09.187 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.181+0000 7fe96e864640 1 -- start start 2023-10-30T17:55:09.187 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.181+0000 7fe96e864640 10 monclient: init 2023-10-30T17:55:09.187 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:09.187 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T17:55:09.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9680b3e20 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:09.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe968144fa0 0x7fe96813ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 10 monclient(hunting): picked mon.noname-c con 0x7fe968144fa0 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:09.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe96813efd0 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:09.188 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680aa950 con 0x7fe96813efd0 2023-10-30T17:55:09.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680aa320 con 0x7fe9680b3e20 2023-10-30T17:55:09.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:09.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe96813f400 con 0x7fe968144fa0 2023-10-30T17:55:09.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:09.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.182+0000 7fe96e864640 10 monclient(hunting): authenticate will time out at 2691.505495s 2023-10-30T17:55:09.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.183+0000 7fe96ce01640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fe96813efd0 0x7fe96814c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59496/0 (socket says 172.21.15.170:59496) 2023-10-30T17:55:09.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.183+0000 7fe96ce01640 1 -- 172.21.15.170:0/3985765425 learned_addr learned my addr 172.21.15.170:0/3985765425 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:09.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.183+0000 7fe9677fe640 1 --2- 172.21.15.170:0/3985765425 >> v2:172.21.15.177:3300/0 conn(0x7fe968144fa0 0x7fe96813ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:09.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.184+0000 7fe9677fe640 10 monclient(hunting): get_auth_request con 0x7fe968144fa0 auth_method 0 2023-10-30T17:55:09.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.184+0000 7fe9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:09.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.184+0000 7fe9677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:09.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.184+0000 7fe9677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:09.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.184+0000 7fe9677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:09.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.184+0000 7fe9677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:09.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.184+0000 7fe9677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:09.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.185+0000 7fe966ffd640 1 -- 172.21.15.170:0/3985765425 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343754427 0 0) 0x7fe9680aa950 con 0x7fe96813efd0 2023-10-30T17:55:09.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.185+0000 7fe966ffd640 10 monclient(hunting): _negotiate 2023-10-30T17:55:09.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.185+0000 7fe966ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:09.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.185+0000 7fe966ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:09.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.185+0000 7fe966ffd640 10 monclient(hunting): my global_id is 16062 2023-10-30T17:55:09.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.185+0000 7fe966ffd640 1 -- 172.21.15.170:0/3985765425 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe950003520 con 0x7fe96813efd0 2023-10-30T17:55:09.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.187+0000 7fe9677fe640 10 monclient(hunting): handle_auth_done global_id 21508 payload 274 2023-10-30T17:55:09.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.187+0000 7fe9677fe640 1 -- 172.21.15.170:0/3985765425 >> v1:172.21.15.177:6789/0 conn(0x7fe9680b3e20 legacy=0x7fe9680a8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.187+0000 7fe9677fe640 1 -- 172.21.15.170:0/3985765425 >> v1:172.21.15.170:6789/0 conn(0x7fe96813efd0 legacy=0x7fe96814c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.187+0000 7fe9677fe640 10 monclient: _finish_hunting 0 2023-10-30T17:55:09.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.187+0000 7fe9677fe640 1 monclient: found mon.noname-c 2023-10-30T17:55:09.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.187+0000 7fe9677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:09.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.187+0000 7fe9677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:55:09.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.187+0000 7fe9677fe640 1 -- 172.21.15.170:0/3985765425 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96813f560 con 0x7fe968144fa0 2023-10-30T17:55:09.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.190+0000 7fe9677fe640 1 --2- 172.21.15.170:0/3985765425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe968144fa0 0x7fe96813ea90 secure :-1 s=READY pgs=5288 cs=0 l=1 rev1=1 crypto rx=0x7fe95c0097a0 tx=0x7fe95c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a361071c60b216de server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:09.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 1 -- 172.21.15.170:0/3985765425 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c02e070 con 0x7fe968144fa0 2023-10-30T17:55:09.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:09.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:09.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 10 monclient: dump: 2023-10-30T17:55:09.197 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:09.197 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:09.197 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.197 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.197 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:09.198 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.198 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:09.198 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:09.198 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:09.198 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 10 monclient: _finish_auth 0 2023-10-30T17:55:09.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 10 monclient: _check_auth_tickets 2023-10-30T17:55:09.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:09.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 1 -- 172.21.15.170:0/3985765425 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c0315a0 con 0x7fe968144fa0 2023-10-30T17:55:09.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe96e864640 5 monclient: authenticate success, global_id 21508 2023-10-30T17:55:09.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe96e864640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:09.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:09.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe96e864640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:09.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe96e864640 4 set_mon_vals no callback set 2023-10-30T17:55:09.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 1 -- 172.21.15.170:0/3985765425 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c031930 con 0x7fe968144fa0 2023-10-30T17:55:09.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:09.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:09.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.194+0000 7fe966ffd640 10 monclient: dump: 2023-10-30T17:55:09.201 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:09.201 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:09.201 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.202 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.202 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:09.202 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.202 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:09.202 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:09.203 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:09.203 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.195+0000 7fe96e864640 10 monclient: shutdown 2023-10-30T17:55:09.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.195+0000 7fe96e864640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:09.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.195+0000 7fe96e864640 1 -- 172.21.15.170:0/3985765425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe968144fa0 msgr2=0x7fe96813ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.206 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.195+0000 7fe96e864640 1 --2- 172.21.15.170:0/3985765425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe968144fa0 0x7fe96813ea90 secure :-1 s=READY pgs=5288 cs=0 l=1 rev1=1 crypto rx=0x7fe95c0097a0 tx=0x7fe95c02b840 comp rx=0 tx=0).stop 2023-10-30T17:55:09.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.195+0000 7fe96e864640 1 -- 172.21.15.170:0/3985765425 shutdown_connections 2023-10-30T17:55:09.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.195+0000 7fe96e864640 1 --2- 172.21.15.170:0/3985765425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe968144fa0 0x7fe96813ea90 unknown :-1 s=CLOSED pgs=5288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.195+0000 7fe96e864640 1 -- 172.21.15.170:0/3985765425 >> 172.21.15.170:0/3985765425 conn(0x7fe968071770 msgr2=0x7fe968071b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:09.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.195+0000 7fe96e864640 1 -- 172.21.15.170:0/3985765425 shutdown_connections 2023-10-30T17:55:09.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.195+0000 7fe96e864640 1 -- 172.21.15.170:0/3985765425 wait complete. 2023-10-30T17:55:09.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.196+0000 7fe96e864640 10 monclient: build_initial_monmap 2023-10-30T17:55:09.208 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.196+0000 7fe96e864640 10 monclient: monmap: 2023-10-30T17:55:09.208 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:09.208 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:09.208 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:09.209 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:09.209 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:09.209 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.209 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:09.209 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:09.210 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:09.210 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.196+0000 7fe96e864640 1 librados: starting msgr at 2023-10-30T17:55:09.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.196+0000 7fe96e864640 1 librados: starting objecter 2023-10-30T17:55:09.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.196+0000 7fe96e864640 1 Processor -- start 2023-10-30T17:55:09.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.196+0000 7fe96e864640 1 -- start start 2023-10-30T17:55:09.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.196+0000 7fe96e864640 1 librados: setting wanted keys 2023-10-30T17:55:09.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.196+0000 7fe96e864640 1 librados: calling monclient init 2023-10-30T17:55:09.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.196+0000 7fe96e864640 10 monclient: init 2023-10-30T17:55:09.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:09.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:55:09.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9680b3e20 0x7fe968142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 10 monclient(hunting): picked mon.noname-b con 0x7fe9680b3e20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:09.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe96813efd0 0x7fe9680ad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 10 monclient(hunting): picked mon.noname-a con 0x7fe96813efd0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:09.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe968144fa0 0x7fe9681d6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 10 monclient(hunting): picked mon.noname-c con 0x7fe968144fa0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:09.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:09.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680ad0c0 con 0x7fe96813efd0 2023-10-30T17:55:09.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:09.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe968140fb0 con 0x7fe9680b3e20 2023-10-30T17:55:09.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:09.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680acc70 con 0x7fe968144fa0 2023-10-30T17:55:09.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:09.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.197+0000 7fe96e864640 10 monclient(hunting): authenticate will time out at 2691.520559s 2023-10-30T17:55:09.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe967fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9680b3e20 0x7fe968142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:09.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe96ce01640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe968144fa0 0x7fe9681d6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:09.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe967fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9680b3e20 0x7fe968142280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:49660/0 (socket says 172.21.15.170:49660) 2023-10-30T17:55:09.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe967fff640 1 -- 172.21.15.170:0/826042650 learned_addr learned my addr 172.21.15.170:0/826042650 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:09.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe967fff640 10 monclient(hunting): get_auth_request con 0x7fe9680b3e20 auth_method 0 2023-10-30T17:55:09.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe967fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:09.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe967fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:09.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe967fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:09.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe96ce01640 10 monclient(hunting): get_auth_request con 0x7fe968144fa0 auth_method 0 2023-10-30T17:55:09.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe96ce01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:09.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe96ce01640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:09.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe96ce01640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:09.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe967fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:09.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe967fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:09.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe967fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:09.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe96ce01640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:09.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.198+0000 7fe96ce01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:09.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.199+0000 7fe96ce01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:09.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.201+0000 7fe967fff640 10 monclient(hunting): handle_auth_done global_id 21514 payload 824 2023-10-30T17:55:09.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.201+0000 7fe967fff640 1 -- 172.21.15.170:0/826042650 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe968144fa0 msgr2=0x7fe9681d6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.201+0000 7fe967fff640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe968144fa0 0x7fe9681d6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.201+0000 7fe967fff640 1 -- 172.21.15.170:0/826042650 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe96813efd0 msgr2=0x7fe9680ad4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.201+0000 7fe967fff640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe96813efd0 0x7fe9680ad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.201+0000 7fe967fff640 10 monclient: _finish_hunting 0 2023-10-30T17:55:09.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.201+0000 7fe967fff640 1 monclient: found mon.noname-b 2023-10-30T17:55:09.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.201+0000 7fe967fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:09.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.201+0000 7fe967fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:09.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.201+0000 7fe967fff640 1 -- 172.21.15.170:0/826042650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c009400 con 0x7fe9680b3e20 2023-10-30T17:55:09.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.203+0000 7fe967fff640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9680b3e20 0x7fe968142280 secure :-1 s=READY pgs=5289 cs=0 l=1 rev1=1 crypto rx=0x7fe95800c450 tx=0x7fe95800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:09.222 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.203+0000 7fe964ff9640 10 client.?.objecter ms_handle_connect 0x7fe9680b3e20 2023-10-30T17:55:09.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.203+0000 7fe964ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:09.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 1 -- 172.21.15.170:0/826042650 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9580052d0 con 0x7fe9680b3e20 2023-10-30T17:55:09.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:09.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:09.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 10 monclient: dump: 2023-10-30T17:55:09.225 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:09.225 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:09.225 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.225 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.226 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:09.226 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.226 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:09.226 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:09.226 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:09.227 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 10 monclient: _finish_auth 0 2023-10-30T17:55:09.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 10 monclient: _check_auth_tickets 2023-10-30T17:55:09.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:09.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 1 -- 172.21.15.170:0/826042650 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe958005430 con 0x7fe9680b3e20 2023-10-30T17:55:09.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:09.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 1 -- 172.21.15.170:0/826042650 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe958012870 con 0x7fe9680b3e20 2023-10-30T17:55:09.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:09.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:09.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe964ff9640 10 monclient: dump: 2023-10-30T17:55:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:09.230 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe96e864640 5 monclient: authenticate success, global_id 21514 2023-10-30T17:55:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe96e864640 10 monclient: _renew_subs 2023-10-30T17:55:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe96e864640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9681413d0 con 0x7fe9680b3e20 2023-10-30T17:55:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe9667fc640 4 set_mon_vals no callback set 2023-10-30T17:55:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe96e864640 10 client.21514.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe96e864640 10 monclient: _renew_subs 2023-10-30T17:55:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe96e864640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9680ada20 con 0x7fe9680b3e20 2023-10-30T17:55:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.207+0000 7fe96e864640 1 librados: init done 2023-10-30T17:55:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.208+0000 7fe96e864640 10 client.21514.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:09.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.208+0000 7fe96e864640 15 client.21514.objecter _session_command_op_assign -1 1 2023-10-30T17:55:09.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.208+0000 7fe96e864640 20 client.21514.objecter _get_session osd=-1 returning homeless 2023-10-30T17:55:09.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.208+0000 7fe96e864640 10 client.21514.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:09.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.208+0000 7fe96e864640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:09.233 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.208+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe9680adec0 con 0x7fe9680b3e20 2023-10-30T17:55:09.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.226+0000 7fe964ff9640 1 -- 172.21.15.170:0/826042650 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe9580129d0 con 0x7fe9680b3e20 2023-10-30T17:55:09.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.226+0000 7fe964ff9640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fe95406c850 0x7fe95406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.227+0000 7fe9677fe640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fe95406c850 0x7fe95406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:09.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.227+0000 7fe9677fe640 10 monclient: get_auth_request con 0x7fe95406c850 auth_method 0 2023-10-30T17:55:09.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.227+0000 7fe964ff9640 10 client.21514.objecter ms_handle_connect 0x7fe95406c850 2023-10-30T17:55:09.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.227+0000 7fe9677fe640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fe95406c850 0x7fe95406ed10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe95c036000 tx=0x7fe95c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:09.237 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 1 -- 172.21.15.170:0/826042650 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fe958059000 con 0x7fe9680b3e20 2023-10-30T17:55:09.237 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 10 client.21514.objecter ms_dispatch 0x7fe968000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:09.237 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 3 client.21514.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:09.237 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 3 client.21514.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:09.237 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 10 client.21514.objecter checking command 1 2023-10-30T17:55:09.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] conn(0x7fe954074060 0x7fe954076500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 20 client.21514.objecter _get_session s=0x7fe954073990 osd=6 3 2023-10-30T17:55:09.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 20 client.21514.objecter put_session s=0x7fe954073990 osd=6 3 2023-10-30T17:55:09.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 15 client.21514.objecter _session_command_op_remove -1 1 2023-10-30T17:55:09.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 20 client.21514.objecter _get_session s=0x7fe954073990 osd=6 3 2023-10-30T17:55:09.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 20 client.21514.objecter get_session s=0x7fe954073990 osd=6 3 2023-10-30T17:55:09.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 15 client.21514.objecter _session_command_op_assign 6 1 2023-10-30T17:55:09.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 20 client.21514.objecter put_session s=0x7fe954073990 osd=6 4 2023-10-30T17:55:09.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 10 client.21514.objecter _send_command 1 2023-10-30T17:55:09.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 1 -- 172.21.15.170:0/826042650 --> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe954076bd0 con 0x7fe954074060 2023-10-30T17:55:09.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe964ff9640 20 client.21514.objecter dump_active .. 0 homeless 2023-10-30T17:55:09.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.229+0000 7fe96ce01640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] conn(0x7fe954074060 0x7fe954076500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:09.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.230+0000 7fe96ce01640 10 monclient: get_auth_request con 0x7fe954074060 auth_method 0 2023-10-30T17:55:09.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.230+0000 7fe964ff9640 1 -- 172.21.15.170:0/826042650 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=914) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9580011e0 con 0x7fe9680b3e20 2023-10-30T17:55:09.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.230+0000 7fe964ff9640 10 monclient: handle_get_version_reply finishing 1 version 914 2023-10-30T17:55:09.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.232+0000 7fe964ff9640 10 client.21514.objecter ms_handle_connect 0x7fe954074060 2023-10-30T17:55:09.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.232+0000 7fe96ce01640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] conn(0x7fe954074060 0x7fe954076500 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:09.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.259+0000 7fe964ff9640 1 -- 172.21.15.170:0/826042650 <== osd.6 v2:172.21.15.177:6826/1041798931 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fe954076bd0 con 0x7fe954074060 2023-10-30T17:55:09.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.259+0000 7fe964ff9640 10 client.21514.objecter ms_dispatch 0x7fe968000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:55:09.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.259+0000 7fe964ff9640 10 client.21514.objecter _finish_command 1 = system:0 2023-10-30T17:55:09.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.259+0000 7fe964ff9640 20 client.21514.objecter put_session s=0x7fe954073990 osd=6 4 2023-10-30T17:55:09.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.259+0000 7fe964ff9640 15 client.21514.objecter _session_command_op_remove 6 1 2023-10-30T17:55:09.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 10 client.21514.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T17:55:09.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 15 client.21514.objecter _session_command_op_assign -1 2 2023-10-30T17:55:09.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 20 client.21514.objecter _get_session s=0x7fe954073990 osd=6 3 2023-10-30T17:55:09.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 20 client.21514.objecter put_session s=0x7fe954073990 osd=6 3 2023-10-30T17:55:09.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 20 client.21514.objecter _get_session s=0x7fe954073990 osd=6 3 2023-10-30T17:55:09.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 15 client.21514.objecter _session_command_op_remove -1 2 2023-10-30T17:55:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 20 client.21514.objecter get_session s=0x7fe954073990 osd=6 3 2023-10-30T17:55:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 15 client.21514.objecter _session_command_op_assign 6 2 2023-10-30T17:55:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 20 client.21514.objecter put_session s=0x7fe954073990 osd=6 4 2023-10-30T17:55:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 10 client.21514.objecter _send_command 2 2023-10-30T17:55:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.279+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 --> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe9680b4200 con 0x7fe954074060 2023-10-30T17:55:09.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.293+0000 7fe964ff9640 1 -- 172.21.15.170:0/826042650 <== osd.6 v2:172.21.15.177:6826/1041798931 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe960002010 con 0x7fe954074060 2023-10-30T17:55:09.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.293+0000 7fe964ff9640 10 client.21514.objecter ms_dispatch 0x7fe968000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:55:09.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.293+0000 7fe964ff9640 10 client.21514.objecter _finish_command 2 = system:0 2023-10-30T17:55:09.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.293+0000 7fe964ff9640 20 client.21514.objecter put_session s=0x7fe954073990 osd=6 4 2023-10-30T17:55:09.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.293+0000 7fe964ff9640 15 client.21514.objecter _session_command_op_remove 6 2 2023-10-30T17:55:09.296 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-30T17:55:09.296 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T17:55:09.296 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-30T17:55:09.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 10 librados: watch_flush enter 2023-10-30T17:55:09.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 10 librados: watch_flush exit 2023-10-30T17:55:09.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 10 client.21514.objecter close_session for osd.6 2023-10-30T17:55:09.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] conn(0x7fe954074060 msgr2=0x7fe954076500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:6826/1041798931,v1:172.21.15.177:6827/1041798931] conn(0x7fe954074060 0x7fe954076500 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 20 client.21514.objecter put_session s=0x7fe954073990 osd=6 1 2023-10-30T17:55:09.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 20 client.21514.objecter shutdown clearing up homeless session... 2023-10-30T17:55:09.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 10 client.21514.objecter successfully canceled tick 2023-10-30T17:55:09.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fe95406c850 msgr2=0x7fe95406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fe95406c850 0x7fe95406ed10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe95c036000 tx=0x7fe95c030000 comp rx=0 tx=0).stop 2023-10-30T17:55:09.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 10 monclient: shutdown 2023-10-30T17:55:09.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:09.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9680b3e20 msgr2=0x7fe968142280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 --2- 172.21.15.170:0/826042650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9680b3e20 0x7fe968142280 secure :-1 s=READY pgs=5289 cs=0 l=1 rev1=1 crypto rx=0x7fe95800c450 tx=0x7fe95800c910 comp rx=0 tx=0).stop 2023-10-30T17:55:09.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96ce01640 1 -- 172.21.15.170:0/826042650 reap_dead start 2023-10-30T17:55:09.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 shutdown_connections 2023-10-30T17:55:09.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 >> 172.21.15.170:0/826042650 conn(0x7fe968071770 msgr2=0x7fe9680a8560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:09.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 shutdown_connections 2023-10-30T17:55:09.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 -- 172.21.15.170:0/826042650 wait complete. 2023-10-30T17:55:09.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.294+0000 7fe96e864640 1 librados: shutdown 2023-10-30T17:55:09.309 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-30T17:55:09.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.412+0000 7fa64e1c9640 10 monclient: get_monmap_and_config 2023-10-30T17:55:09.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.412+0000 7fa64e1c9640 10 monclient: build_initial_monmap 2023-10-30T17:55:09.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.412+0000 7fa64e1c9640 10 monclient: monmap: 2023-10-30T17:55:09.414 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:09.414 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:09.415 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:09.413230+0000 2023-10-30T17:55:09.415 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:09.413230+0000 2023-10-30T17:55:09.415 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:09.415 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.415 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:09.416 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:09.416 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:09.416 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:09.416 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:09.416 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 1 Processor -- start 2023-10-30T17:55:09.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 1 -- start start 2023-10-30T17:55:09.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 10 monclient: init 2023-10-30T17:55:09.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:09.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-10-30T17:55:09.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa6480b7c90 0x7fa6480a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 10 monclient(hunting): picked mon.noname-a con 0x7fa6480b7c90 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:09.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa64813eb60 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:09.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa648144f50 0x7fa648145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 10 monclient(hunting): picked mon.noname-b con 0x7fa648144f50 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:09.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6480aa740 con 0x7fa64813eb60 2023-10-30T17:55:09.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:09.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6480a9500 con 0x7fa6480b7c90 2023-10-30T17:55:09.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:09.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64813f6c0 con 0x7fa648144f50 2023-10-30T17:55:09.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:09.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.413+0000 7fa64e1c9640 10 monclient(hunting): authenticate will time out at 2691.736768s 2023-10-30T17:55:09.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.414+0000 7fa647fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa648144f50 0x7fa648145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:09.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.414+0000 7fa647fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa648144f50 0x7fa648145400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:39142/0 (socket says 172.21.15.170:39142) 2023-10-30T17:55:09.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.414+0000 7fa647fff640 1 -- 172.21.15.170:0/2824238345 learned_addr learned my addr 172.21.15.170:0/2824238345 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:09.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.414+0000 7fa647fff640 10 monclient(hunting): get_auth_request con 0x7fa648144f50 auth_method 0 2023-10-30T17:55:09.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.414+0000 7fa647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:09.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.414+0000 7fa647fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:09.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.414+0000 7fa647fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:09.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.415+0000 7fa647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:09.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.415+0000 7fa647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:09.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.415+0000 7fa647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:09.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.416+0000 7fa647fff640 10 monclient(hunting): handle_auth_done global_id 16373 payload 274 2023-10-30T17:55:09.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.416+0000 7fa647fff640 1 -- 172.21.15.170:0/2824238345 >> v2:172.21.15.170:3300/0 conn(0x7fa6480b7c90 msgr2=0x7fa6480a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:09.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.416+0000 7fa647fff640 1 --2- 172.21.15.170:0/2824238345 >> v2:172.21.15.170:3300/0 conn(0x7fa6480b7c90 0x7fa6480a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.416+0000 7fa647fff640 1 -- 172.21.15.170:0/2824238345 >> v1:172.21.15.170:6789/0 conn(0x7fa64813eb60 legacy=0x7fa64813efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:09.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.416+0000 7fa647fff640 10 monclient: _finish_hunting 0 2023-10-30T17:55:09.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.416+0000 7fa647fff640 1 monclient: found mon.noname-b 2023-10-30T17:55:09.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.416+0000 7fa647fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:09.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.416+0000 7fa647fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.170:3301/0 2023-10-30T17:55:09.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.416+0000 7fa647fff640 1 -- 172.21.15.170:0/2824238345 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6480aa9b0 con 0x7fa648144f50 2023-10-30T17:55:09.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.418+0000 7fa647fff640 1 --2- 172.21.15.170:0/2824238345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa648144f50 0x7fa648145400 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fa640009e50 tx=0x7fa64002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bf8179a09cd0c0b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:09.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 1 -- 172.21.15.170:0/2824238345 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64002c9e0 con 0x7fa648144f50 2023-10-30T17:55:09.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:09.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:09.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 10 monclient: dump: 2023-10-30T17:55:09.434 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:09.434 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:09.435 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.435 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.435 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:09.435 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.435 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:09.435 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:09.436 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:09.436 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 10 monclient: _finish_auth 0 2023-10-30T17:55:09.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 10 monclient: _check_auth_tickets 2023-10-30T17:55:09.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:09.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 1 -- 172.21.15.170:0/2824238345 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64002cb40 con 0x7fa648144f50 2023-10-30T17:55:09.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa64e1c9640 5 monclient: authenticate success, global_id 16373 2023-10-30T17:55:09.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa64e1c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:09.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:09.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa64e1c9640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:09.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 1 -- 172.21.15.170:0/2824238345 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6400343f0 con 0x7fa648144f50 2023-10-30T17:55:09.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:09.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa64e1c9640 4 set_mon_vals no callback set 2023-10-30T17:55:09.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:55:09.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.422+0000 7fa6467fc640 10 monclient: dump: 2023-10-30T17:55:09.439 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:09.439 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:09.440 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.440 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.440 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:09.440 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.440 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:09.441 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:09.441 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:09.441 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.423+0000 7fa64e1c9640 10 monclient: shutdown 2023-10-30T17:55:09.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.423+0000 7fa64e1c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:09.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.423+0000 7fa64e1c9640 1 -- 172.21.15.170:0/2824238345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa648144f50 msgr2=0x7fa648145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.423+0000 7fa64e1c9640 1 --2- 172.21.15.170:0/2824238345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa648144f50 0x7fa648145400 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fa640009e50 tx=0x7fa64002be40 comp rx=0 tx=0).stop 2023-10-30T17:55:09.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.423+0000 7fa64e1c9640 1 -- 172.21.15.170:0/2824238345 shutdown_connections 2023-10-30T17:55:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.423+0000 7fa64e1c9640 1 --2- 172.21.15.170:0/2824238345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa648144f50 0x7fa648145400 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.423+0000 7fa64e1c9640 1 --2- 172.21.15.170:0/2824238345 >> v2:172.21.15.170:3300/0 conn(0x7fa6480b7c90 0x7fa6480a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.423+0000 7fa64e1c9640 1 -- 172.21.15.170:0/2824238345 >> 172.21.15.170:0/2824238345 conn(0x7fa648071da0 msgr2=0x7fa6480b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.423+0000 7fa64e1c9640 1 -- 172.21.15.170:0/2824238345 shutdown_connections 2023-10-30T17:55:09.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.423+0000 7fa64e1c9640 1 -- 172.21.15.170:0/2824238345 wait complete. 2023-10-30T17:55:09.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.424+0000 7fa64e1c9640 10 monclient: build_initial_monmap 2023-10-30T17:55:09.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.424+0000 7fa64e1c9640 10 monclient: monmap: 2023-10-30T17:55:09.444 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:09.445 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:09.445 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:09.445 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:09.445 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:09.445 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.446 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:09.446 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:09.446 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:09.447 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.424+0000 7fa64e1c9640 1 librados: starting msgr at 2023-10-30T17:55:09.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.424+0000 7fa64e1c9640 1 librados: starting objecter 2023-10-30T17:55:09.447 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.425+0000 7fa64e1c9640 1 Processor -- start 2023-10-30T17:55:09.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.425+0000 7fa64e1c9640 1 -- start start 2023-10-30T17:55:09.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.425+0000 7fa64e1c9640 1 librados: setting wanted keys 2023-10-30T17:55:09.448 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.425+0000 7fa64e1c9640 1 librados: calling monclient init 2023-10-30T17:55:09.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.425+0000 7fa64e1c9640 10 monclient: init 2023-10-30T17:55:09.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:09.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T17:55:09.449 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6480b7c90 0x7fa648140510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 10 monclient(hunting): picked mon.noname-b con 0x7fa6480b7c90 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:09.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64813eb60 0x7fa648140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 10 monclient(hunting): picked mon.noname-c con 0x7fa64813eb60 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:09.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa648144f50 0x7fa648135600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 10 monclient(hunting): picked mon.noname-a con 0x7fa648144f50 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:09.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:09.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64813a2b0 con 0x7fa648144f50 2023-10-30T17:55:09.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:09.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa648140f90 con 0x7fa6480b7c90 2023-10-30T17:55:09.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:09.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6481426b0 con 0x7fa64813eb60 2023-10-30T17:55:09.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:09.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa64e1c9640 10 monclient(hunting): authenticate will time out at 2691.749339s 2023-10-30T17:55:09.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.426+0000 7fa6477fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6480b7c90 0x7fa648140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:09.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.427+0000 7fa6477fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6480b7c90 0x7fa648140510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:49670/0 (socket says 172.21.15.170:49670) 2023-10-30T17:55:09.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.427+0000 7fa6477fe640 1 -- 172.21.15.170:0/1206947119 learned_addr learned my addr 172.21.15.170:0/1206947119 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:09.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.427+0000 7fa6477fe640 10 monclient(hunting): get_auth_request con 0x7fa6480b7c90 auth_method 0 2023-10-30T17:55:09.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.427+0000 7fa6477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:09.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.427+0000 7fa6477fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:09.454 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.427+0000 7fa6477fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:09.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.427+0000 7fa646ffd640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64813eb60 0x7fa648140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:09.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.427+0000 7fa6477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:09.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.427+0000 7fa6477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:09.455 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.427+0000 7fa6477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:09.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.428+0000 7fa646ffd640 10 monclient(hunting): get_auth_request con 0x7fa64813eb60 auth_method 0 2023-10-30T17:55:09.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.428+0000 7fa646ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:09.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.428+0000 7fa646ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:09.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.428+0000 7fa646ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:09.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.428+0000 7fa646ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:09.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.428+0000 7fa646ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:09.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.428+0000 7fa646ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:09.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.430+0000 7fa6477fe640 10 monclient(hunting): handle_auth_done global_id 21517 payload 824 2023-10-30T17:55:09.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.430+0000 7fa6477fe640 1 -- 172.21.15.170:0/1206947119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64813eb60 msgr2=0x7fa648140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.430+0000 7fa6477fe640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64813eb60 0x7fa648140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.430+0000 7fa6477fe640 1 -- 172.21.15.170:0/1206947119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa648144f50 msgr2=0x7fa648135600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.430+0000 7fa6477fe640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa648144f50 0x7fa648135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.430+0000 7fa6477fe640 10 monclient: _finish_hunting 0 2023-10-30T17:55:09.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.430+0000 7fa6477fe640 1 monclient: found mon.noname-b 2023-10-30T17:55:09.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.430+0000 7fa6477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:09.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.430+0000 7fa6477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:09.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.430+0000 7fa6477fe640 1 -- 172.21.15.170:0/1206947119 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa640009b30 con 0x7fa6480b7c90 2023-10-30T17:55:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.432+0000 7fa6477fe640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6480b7c90 0x7fa648140510 secure :-1 s=READY pgs=5290 cs=0 l=1 rev1=1 crypto rx=0x7fa63800b4a0 tx=0x7fa63800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.432+0000 7fa62ffff640 10 client.?.objecter ms_handle_connect 0x7fa6480b7c90 2023-10-30T17:55:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.432+0000 7fa62ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 1 -- 172.21.15.170:0/1206947119 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa638010610 con 0x7fa6480b7c90 2023-10-30T17:55:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:09.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:09.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 10 monclient: dump: 2023-10-30T17:55:09.462 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:09.462 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:09.464 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:09.464 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 10 monclient: _finish_auth 0 2023-10-30T17:55:09.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 10 monclient: _check_auth_tickets 2023-10-30T17:55:09.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:09.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 1 -- 172.21.15.170:0/1206947119 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa638010770 con 0x7fa6480b7c90 2023-10-30T17:55:09.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:09.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 1 -- 172.21.15.170:0/1206947119 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa638010b20 con 0x7fa6480b7c90 2023-10-30T17:55:09.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:09.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:09.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa62ffff640 10 monclient: dump: 2023-10-30T17:55:09.466 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:09.466 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:09.466 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.466 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:09.466 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:09.467 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:09.467 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:09.467 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:09.467 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:09.467 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:09.468 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa64e1c9640 5 monclient: authenticate success, global_id 21517 2023-10-30T17:55:09.468 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa64e1c9640 10 monclient: _renew_subs 2023-10-30T17:55:09.468 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa64e1c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:09.468 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa64e1c9640 1 -- 172.21.15.170:0/1206947119 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa648142990 con 0x7fa6480b7c90 2023-10-30T17:55:09.468 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa645ffb640 4 set_mon_vals no callback set 2023-10-30T17:55:09.469 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa64e1c9640 10 client.21517.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:09.469 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa64e1c9640 10 monclient: _renew_subs 2023-10-30T17:55:09.469 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa64e1c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:09.469 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa64e1c9640 1 -- 172.21.15.170:0/1206947119 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa648135d20 con 0x7fa6480b7c90 2023-10-30T17:55:09.469 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.436+0000 7fa64e1c9640 1 librados: init done 2023-10-30T17:55:09.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.437+0000 7fa64e1c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:09.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.437+0000 7fa64e1c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:09.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.437+0000 7fa64e1c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:09.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.437+0000 7fa64e1c9640 1 -- 172.21.15.170:0/1206947119 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6481423f0 con 0x7fa6480b7c90 2023-10-30T17:55:09.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.455+0000 7fa62ffff640 1 -- 172.21.15.170:0/1206947119 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa638010c80 con 0x7fa6480b7c90 2023-10-30T17:55:09.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.456+0000 7fa62ffff640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa63406c880 0x7fa63406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:09.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.456+0000 7fa646ffd640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa63406c880 0x7fa63406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:09.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.456+0000 7fa646ffd640 10 monclient: get_auth_request con 0x7fa63406c880 auth_method 0 2023-10-30T17:55:09.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.457+0000 7fa646ffd640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa63406c880 0x7fa63406ed40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa6481d4130 tx=0x7fa63c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:09.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.457+0000 7fa62ffff640 10 client.21517.objecter ms_handle_connect 0x7fa63406c880 2023-10-30T17:55:09.472 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:53.432 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:09.473 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:53.434 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:09.497 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:53.463 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:09.528 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:53.494 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:09.535 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:53.501 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:09.564 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:53.530 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:09.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.683+0000 7fa62ffff640 1 -- 172.21.15.170:0/1206947119 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(914..914 src has 374..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fa638058230 con 0x7fa6480b7c90 2023-10-30T17:55:09.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.683+0000 7fa62ffff640 10 client.21517.objecter ms_dispatch 0x7fa648000b90 osd_map(914..914 src has 374..914) v4 2023-10-30T17:55:09.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.683+0000 7fa62ffff640 3 client.21517.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:09.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.683+0000 7fa62ffff640 3 client.21517.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:09.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.683+0000 7fa62ffff640 20 client.21517.objecter dump_active .. 0 homeless 2023-10-30T17:55:09.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.692+0000 7fa62ffff640 1 -- 172.21.15.170:0/1206947119 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa638054570 con 0x7fa6480b7c90 2023-10-30T17:55:09.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.692+0000 7fa62ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:09.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.692+0000 7fa62ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:09.769 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:53.708 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:09.771 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:53.710 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:09.792 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:53.731 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:09.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.873+0000 7fa64e1c9640 1 -- 172.21.15.170:0/1206947119 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6480a90c0 con 0x7fa63406c880 2023-10-30T17:55:09.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.873+0000 7fa62ffff640 1 -- 172.21.15.170:0/1206947119 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa63c001440 con 0x7fa63406c880 2023-10-30T17:55:09.875 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 6 to deep-scrub 2023-10-30T17:55:09.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.875+0000 7fa64e1c9640 10 librados: watch_flush enter 2023-10-30T17:55:09.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.875+0000 7fa64e1c9640 10 librados: watch_flush exit 2023-10-30T17:55:09.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.875+0000 7fa64e1c9640 20 client.21517.objecter shutdown clearing up homeless session... 2023-10-30T17:55:09.878 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.875+0000 7fa64e1c9640 10 client.21517.objecter successfully canceled tick 2023-10-30T17:55:09.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 -- 172.21.15.170:0/1206947119 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa63406c880 msgr2=0x7fa63406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa63406c880 0x7fa63406ed40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa6481d4130 tx=0x7fa63c009040 comp rx=0 tx=0).stop 2023-10-30T17:55:09.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 10 monclient: shutdown 2023-10-30T17:55:09.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:09.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 -- 172.21.15.170:0/1206947119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6480b7c90 msgr2=0x7fa648140510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:09.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6480b7c90 0x7fa648140510 secure :-1 s=READY pgs=5290 cs=0 l=1 rev1=1 crypto rx=0x7fa63800b4a0 tx=0x7fa63800b960 comp rx=0 tx=0).stop 2023-10-30T17:55:09.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 -- 172.21.15.170:0/1206947119 shutdown_connections 2023-10-30T17:55:09.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fa63406c880 0x7fa63406ed40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa648144f50 0x7fa648135600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64813eb60 0x7fa648140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 --2- 172.21.15.170:0/1206947119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6480b7c90 0x7fa648140510 unknown :-1 s=CLOSED pgs=5290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:09.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 -- 172.21.15.170:0/1206947119 >> 172.21.15.170:0/1206947119 conn(0x7fa648071da0 msgr2=0x7fa6480b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:09.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 -- 172.21.15.170:0/1206947119 shutdown_connections 2023-10-30T17:55:09.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 -- 172.21.15.170:0/1206947119 wait complete. 2023-10-30T17:55:09.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.876+0000 7fa64e1c9640 1 librados: shutdown 2023-10-30T17:55:09.889 INFO:tasks.ceph:Scrubbing osd.7 2023-10-30T17:55:09.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-30T17:55:09.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.998+0000 7ff0b01dc640 10 monclient: get_monmap_and_config 2023-10-30T17:55:10.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.998+0000 7ff0b01dc640 10 monclient: build_initial_monmap 2023-10-30T17:55:10.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.998+0000 7ff0b01dc640 10 monclient: monmap: 2023-10-30T17:55:10.000 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:10.000 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.000 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:09.999469+0000 2023-10-30T17:55:10.001 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:09.999469+0000 2023-10-30T17:55:10.001 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:10.001 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.001 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:10.001 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:10.002 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:10.002 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:10.002 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:10.002 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.999+0000 7ff0b01dc640 1 Processor -- start 2023-10-30T17:55:10.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.999+0000 7ff0b01dc640 1 -- start start 2023-10-30T17:55:10.004 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.999+0000 7ff0b01dc640 10 monclient: init 2023-10-30T17:55:10.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:09.999+0000 7ff0b01dc640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:10.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-10-30T17:55:10.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff0a80b3e20 0x7ff0a80a8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 10 monclient(hunting): picked mon.noname-a con 0x7ff0a80b3e20 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:10.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff0a8144fa0 0x7ff0a813ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 10 monclient(hunting): picked mon.noname-c con 0x7ff0a8144fa0 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:10.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0a813efd0 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:10.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a80aa950 con 0x7ff0a813efd0 2023-10-30T17:55:10.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a813f400 con 0x7ff0a80b3e20 2023-10-30T17:55:10.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a80b4200 con 0x7ff0a8144fa0 2023-10-30T17:55:10.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:10.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.000+0000 7ff0b01dc640 10 monclient(hunting): authenticate will time out at 2692.323270s 2023-10-30T17:55:10.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.002+0000 7ff0adf78640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff0a80b3e20 0x7ff0a80a8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.007 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.002+0000 7ff0ae779640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7ff0a813efd0 0x7ff0a80b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59512/0 (socket says 172.21.15.170:59512) 2023-10-30T17:55:10.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.003+0000 7ff0ae779640 1 -- 172.21.15.170:0/1823552977 learned_addr learned my addr 172.21.15.170:0/1823552977 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:10.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.003+0000 7ff0adf78640 10 monclient(hunting): get_auth_request con 0x7ff0a80b3e20 auth_method 0 2023-10-30T17:55:10.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.003+0000 7ff0adf78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.003+0000 7ff0adf78640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.003+0000 7ff0adf78640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.004+0000 7ff0adf78640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:10.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.004+0000 7ff0adf78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:10.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.004+0000 7ff0adf78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:10.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.006+0000 7ff0adf78640 10 monclient(hunting): handle_auth_done global_id 16065 payload 274 2023-10-30T17:55:10.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.006+0000 7ff0adf78640 1 -- 172.21.15.170:0/1823552977 >> v2:172.21.15.177:3300/0 conn(0x7ff0a8144fa0 msgr2=0x7ff0a813ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:10.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.006+0000 7ff0adf78640 1 --2- 172.21.15.170:0/1823552977 >> v2:172.21.15.177:3300/0 conn(0x7ff0a8144fa0 0x7ff0a813ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.006+0000 7ff0adf78640 1 -- 172.21.15.170:0/1823552977 >> v1:172.21.15.170:6789/0 conn(0x7ff0a813efd0 legacy=0x7ff0a80b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.006+0000 7ff0adf78640 10 monclient: _finish_hunting 0 2023-10-30T17:55:10.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.006+0000 7ff0adf78640 1 monclient: found mon.noname-a 2023-10-30T17:55:10.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.006+0000 7ff0adf78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:10.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.006+0000 7ff0adf78640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:10.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.006+0000 7ff0adf78640 1 -- 172.21.15.170:0/1823552977 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a80a8ba0 con 0x7ff0a80b3e20 2023-10-30T17:55:10.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.010+0000 7ff0adf78640 1 --2- 172.21.15.170:0/1823552977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0a80b3e20 0x7ff0a80a8660 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80a9c10 tx=0x7ff0a402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe45f5674ce2b0d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:10.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 1 -- 172.21.15.170:0/1823552977 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4031710 con 0x7ff0a80b3e20 2023-10-30T17:55:10.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:10.031 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 10 monclient: dump: 2023-10-30T17:55:10.032 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.032 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.032 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.032 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.032 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.032 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.033 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.033 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.033 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.033 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 10 monclient: _finish_auth 0 2023-10-30T17:55:10.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 10 monclient: _check_auth_tickets 2023-10-30T17:55:10.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:10.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 1 -- 172.21.15.170:0/1823552977 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a402f050 con 0x7ff0a80b3e20 2023-10-30T17:55:10.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:10.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 1 -- 172.21.15.170:0/1823552977 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4031a90 con 0x7ff0a80b3e20 2023-10-30T17:55:10.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:10.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0acf76640 10 monclient: dump: 2023-10-30T17:55:10.035 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.035 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.036 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.036 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.036 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.036 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.036 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.036 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.037 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.037 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.037 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0b01dc640 5 monclient: authenticate success, global_id 16065 2023-10-30T17:55:10.037 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0b01dc640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:10.037 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.029+0000 7ff0b01dc640 4 set_mon_vals no callback set 2023-10-30T17:55:10.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.030+0000 7ff0b01dc640 10 monclient: shutdown 2023-10-30T17:55:10.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.030+0000 7ff0b01dc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:10.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.030+0000 7ff0b01dc640 1 -- 172.21.15.170:0/1823552977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0a80b3e20 msgr2=0x7ff0a80a8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.030+0000 7ff0b01dc640 1 --2- 172.21.15.170:0/1823552977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0a80b3e20 0x7ff0a80a8660 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80a9c10 tx=0x7ff0a402b7b0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.030+0000 7ff0b01dc640 1 -- 172.21.15.170:0/1823552977 shutdown_connections 2023-10-30T17:55:10.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.030+0000 7ff0b01dc640 1 --2- 172.21.15.170:0/1823552977 >> v2:172.21.15.177:3300/0 conn(0x7ff0a8144fa0 0x7ff0a813ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.030+0000 7ff0b01dc640 1 --2- 172.21.15.170:0/1823552977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0a80b3e20 0x7ff0a80a8660 unknown :-1 s=CLOSED pgs=3380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.030+0000 7ff0b01dc640 1 -- 172.21.15.170:0/1823552977 >> 172.21.15.170:0/1823552977 conn(0x7ff0a8071770 msgr2=0x7ff0a8071b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:10.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.030+0000 7ff0b01dc640 1 -- 172.21.15.170:0/1823552977 shutdown_connections 2023-10-30T17:55:10.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.030+0000 7ff0b01dc640 1 -- 172.21.15.170:0/1823552977 wait complete. 2023-10-30T17:55:10.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.031+0000 7ff0b01dc640 10 monclient: build_initial_monmap 2023-10-30T17:55:10.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.031+0000 7ff0b01dc640 10 monclient: monmap: 2023-10-30T17:55:10.043 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:10.043 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:10.043 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:10.045 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:10.045 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:10.045 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.045 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:10.046 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:10.046 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:10.046 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.046 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.032+0000 7ff0b01dc640 1 librados: starting msgr at 2023-10-30T17:55:10.046 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.032+0000 7ff0b01dc640 1 librados: starting objecter 2023-10-30T17:55:10.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.032+0000 7ff0b01dc640 1 Processor -- start 2023-10-30T17:55:10.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.032+0000 7ff0b01dc640 1 -- start start 2023-10-30T17:55:10.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.032+0000 7ff0b01dc640 1 librados: setting wanted keys 2023-10-30T17:55:10.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.032+0000 7ff0b01dc640 1 librados: calling monclient init 2023-10-30T17:55:10.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.032+0000 7ff0b01dc640 10 monclient: init 2023-10-30T17:55:10.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:10.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:55:10.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0a80b3e20 0x7ff0a81447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 10 monclient(hunting): picked mon.noname-c con 0x7ff0a80b3e20 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:10.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a813efd0 0x7ff0a813a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 10 monclient(hunting): picked mon.noname-b con 0x7ff0a813efd0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:10.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0a8144fa0 0x7ff0a813ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 10 monclient(hunting): picked mon.noname-a con 0x7ff0a8144fa0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:10.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a8139e10 con 0x7ff0a8144fa0 2023-10-30T17:55:10.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a813b080 con 0x7ff0a813efd0 2023-10-30T17:55:10.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a8144050 con 0x7ff0a80b3e20 2023-10-30T17:55:10.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:10.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.033+0000 7ff0b01dc640 10 monclient(hunting): authenticate will time out at 2692.356174s 2023-10-30T17:55:10.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.034+0000 7ff0adf78640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0a80b3e20 0x7ff0a81447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.034+0000 7ff0adf78640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0a80b3e20 0x7ff0a81447c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:39160/0 (socket says 172.21.15.170:39160) 2023-10-30T17:55:10.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.034+0000 7ff0adf78640 1 -- 172.21.15.170:0/3950335687 learned_addr learned my addr 172.21.15.170:0/3950335687 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:10.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.034+0000 7ff0adf78640 10 monclient(hunting): get_auth_request con 0x7ff0a80b3e20 auth_method 0 2023-10-30T17:55:10.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.034+0000 7ff0adf78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.051 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.034+0000 7ff0adf78640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.034+0000 7ff0adf78640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.036+0000 7ff0adf78640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:10.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.036+0000 7ff0adf78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:10.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.036+0000 7ff0adf78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:10.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0ae779640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0a8144fa0 0x7ff0a813ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0adf78640 10 monclient(hunting): handle_auth_done global_id 16379 payload 824 2023-10-30T17:55:10.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0adf78640 1 -- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a813efd0 msgr2=0x7ff0a813a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0adf78640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a813efd0 0x7ff0a813a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0adf78640 1 -- 172.21.15.170:0/3950335687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0a8144fa0 msgr2=0x7ff0a813ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0adf78640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0a8144fa0 0x7ff0a813ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0adf78640 10 monclient: _finish_hunting 0 2023-10-30T17:55:10.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0adf78640 1 monclient: found mon.noname-c 2023-10-30T17:55:10.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0adf78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:10.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0adf78640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:55:10.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.039+0000 7ff0adf78640 1 -- 172.21.15.170:0/3950335687 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a4009400 con 0x7ff0a80b3e20 2023-10-30T17:55:10.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.043+0000 7ff0adf78640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0a80b3e20 0x7ff0a81447c0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009890 tx=0x7ff0a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:10.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.043+0000 7ff09e7fc640 10 client.?.objecter ms_handle_connect 0x7ff0a80b3e20 2023-10-30T17:55:10.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.043+0000 7ff09e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:10.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 1 -- 172.21.15.170:0/3950335687 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4033ad0 con 0x7ff0a80b3e20 2023-10-30T17:55:10.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:10.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 10 monclient: dump: 2023-10-30T17:55:10.057 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.057 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.057 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.058 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.058 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.058 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.058 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.058 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.059 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.059 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 10 monclient: _finish_auth 0 2023-10-30T17:55:10.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 10 monclient: _check_auth_tickets 2023-10-30T17:55:10.059 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:10.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 1 -- 172.21.15.170:0/3950335687 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a4033c30 con 0x7ff0a80b3e20 2023-10-30T17:55:10.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff0b01dc640 5 monclient: authenticate success, global_id 16379 2023-10-30T17:55:10.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff0b01dc640 10 monclient: _renew_subs 2023-10-30T17:55:10.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff0b01dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:55:10.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a8142da0 con 0x7ff0a80b3e20 2023-10-30T17:55:10.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:10.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 1 -- 172.21.15.170:0/3950335687 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a403c400 con 0x7ff0a80b3e20 2023-10-30T17:55:10.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:55:10.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09e7fc640 10 monclient: dump: 2023-10-30T17:55:10.061 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.062 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.062 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.062 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.062 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.062 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.062 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.063 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.063 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.063 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff09ffff640 4 set_mon_vals no callback set 2023-10-30T17:55:10.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff0b01dc640 10 client.16379.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:10.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff0b01dc640 10 monclient: _renew_subs 2023-10-30T17:55:10.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff0b01dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:55:10.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a8144d00 con 0x7ff0a80b3e20 2023-10-30T17:55:10.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.049+0000 7ff0b01dc640 1 librados: init done 2023-10-30T17:55:10.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.050+0000 7ff0b01dc640 10 client.16379.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:10.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.050+0000 7ff0b01dc640 15 client.16379.objecter _session_command_op_assign -1 1 2023-10-30T17:55:10.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.050+0000 7ff0b01dc640 20 client.16379.objecter _get_session osd=-1 returning homeless 2023-10-30T17:55:10.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.050+0000 7ff0b01dc640 10 client.16379.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:10.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.050+0000 7ff0b01dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:55:10.065 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.050+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff0a813f560 con 0x7ff0a80b3e20 2023-10-30T17:55:10.073 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.072+0000 7ff09e7fc640 1 -- 172.21.15.170:0/3950335687 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff0a403a040 con 0x7ff0a80b3e20 2023-10-30T17:55:10.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.073+0000 7ff09e7fc640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff08c06c850 0x7ff08c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.073+0000 7ff0ad777640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff08c06c850 0x7ff08c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.073+0000 7ff0ad777640 10 monclient: get_auth_request con 0x7ff08c06c850 auth_method 0 2023-10-30T17:55:10.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.073+0000 7ff09e7fc640 10 client.16379.objecter ms_handle_connect 0x7ff08c06c850 2023-10-30T17:55:10.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.073+0000 7ff0ad777640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff08c06c850 0x7ff08c06ed10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8139920 tx=0x7ff098006bf0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:10.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 1 -- 172.21.15.170:0/3950335687 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(914..914 src has 703..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7ff0a407abb0 con 0x7ff0a80b3e20 2023-10-30T17:55:10.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 10 client.16379.objecter ms_dispatch 0x7ff0a8000b90 osd_map(914..914 src has 703..914) v4 2023-10-30T17:55:10.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 3 client.16379.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:10.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 3 client.16379.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:10.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 10 client.16379.objecter checking command 1 2023-10-30T17:55:10.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] conn(0x7ff08c074060 0x7ff08c076500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 20 client.16379.objecter _get_session s=0x7ff08c073990 osd=7 3 2023-10-30T17:55:10.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 20 client.16379.objecter put_session s=0x7ff08c073990 osd=7 3 2023-10-30T17:55:10.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 15 client.16379.objecter _session_command_op_remove -1 1 2023-10-30T17:55:10.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 20 client.16379.objecter _get_session s=0x7ff08c073990 osd=7 3 2023-10-30T17:55:10.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 20 client.16379.objecter get_session s=0x7ff08c073990 osd=7 3 2023-10-30T17:55:10.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 15 client.16379.objecter _session_command_op_assign 7 1 2023-10-30T17:55:10.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 20 client.16379.objecter put_session s=0x7ff08c073990 osd=7 4 2023-10-30T17:55:10.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 10 client.16379.objecter _send_command 1 2023-10-30T17:55:10.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 1 -- 172.21.15.170:0/3950335687 --> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff08c076bd0 con 0x7ff08c074060 2023-10-30T17:55:10.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 20 client.16379.objecter dump_active .. 0 homeless 2023-10-30T17:55:10.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 1 -- 172.21.15.170:0/3950335687 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_get_version_reply(handle=1 version=914) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0a40767f0 con 0x7ff0a80b3e20 2023-10-30T17:55:10.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.076+0000 7ff09e7fc640 10 monclient: handle_get_version_reply finishing 1 version 914 2023-10-30T17:55:10.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.077+0000 7ff0ae779640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] conn(0x7ff08c074060 0x7ff08c076500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.077+0000 7ff0ae779640 10 monclient: get_auth_request con 0x7ff08c074060 auth_method 0 2023-10-30T17:55:10.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.080+0000 7ff09e7fc640 10 client.16379.objecter ms_handle_connect 0x7ff08c074060 2023-10-30T17:55:10.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.080+0000 7ff0ae779640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] conn(0x7ff08c074060 0x7ff08c076500 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:10.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.103+0000 7ff09e7fc640 1 -- 172.21.15.170:0/3950335687 <== osd.7 v2:172.21.15.177:6818/3145923709 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7ff08c076bd0 con 0x7ff08c074060 2023-10-30T17:55:10.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.103+0000 7ff09e7fc640 10 client.16379.objecter ms_dispatch 0x7ff0a8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T17:55:10.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.103+0000 7ff09e7fc640 10 client.16379.objecter _finish_command 1 = system:0 2023-10-30T17:55:10.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.103+0000 7ff09e7fc640 20 client.16379.objecter put_session s=0x7ff08c073990 osd=7 4 2023-10-30T17:55:10.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.103+0000 7ff09e7fc640 15 client.16379.objecter _session_command_op_remove 7 1 2023-10-30T17:55:10.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 10 client.16379.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T17:55:10.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 15 client.16379.objecter _session_command_op_assign -1 2 2023-10-30T17:55:10.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 20 client.16379.objecter _get_session s=0x7ff08c073990 osd=7 3 2023-10-30T17:55:10.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 20 client.16379.objecter put_session s=0x7ff08c073990 osd=7 3 2023-10-30T17:55:10.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 20 client.16379.objecter _get_session s=0x7ff08c073990 osd=7 3 2023-10-30T17:55:10.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 15 client.16379.objecter _session_command_op_remove -1 2 2023-10-30T17:55:10.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 20 client.16379.objecter get_session s=0x7ff08c073990 osd=7 3 2023-10-30T17:55:10.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 15 client.16379.objecter _session_command_op_assign 7 2 2023-10-30T17:55:10.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 20 client.16379.objecter put_session s=0x7ff08c073990 osd=7 4 2023-10-30T17:55:10.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 10 client.16379.objecter _send_command 2 2023-10-30T17:55:10.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.116+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 --> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff0a80aa360 con 0x7ff08c074060 2023-10-30T17:55:10.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff09e7fc640 1 -- 172.21.15.170:0/3950335687 <== osd.7 v2:172.21.15.177:6818/3145923709 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ff0a00070f0 con 0x7ff08c074060 2023-10-30T17:55:10.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff09e7fc640 10 client.16379.objecter ms_dispatch 0x7ff0a8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T17:55:10.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff09e7fc640 10 client.16379.objecter _finish_command 2 = system:0 2023-10-30T17:55:10.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff09e7fc640 20 client.16379.objecter put_session s=0x7ff08c073990 osd=7 4 2023-10-30T17:55:10.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff09e7fc640 15 client.16379.objecter _session_command_op_remove 7 2 2023-10-30T17:55:10.132 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-30T17:55:10.133 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T17:55:10.133 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-30T17:55:10.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff0b01dc640 10 librados: watch_flush enter 2023-10-30T17:55:10.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff0b01dc640 10 librados: watch_flush exit 2023-10-30T17:55:10.134 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff0b01dc640 10 client.16379.objecter close_session for osd.7 2023-10-30T17:55:10.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] conn(0x7ff08c074060 msgr2=0x7ff08c076500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff0b01dc640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:6818/3145923709,v1:172.21.15.177:6819/3145923709] conn(0x7ff08c074060 0x7ff08c076500 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff0b01dc640 20 client.16379.objecter put_session s=0x7ff08c073990 osd=7 1 2023-10-30T17:55:10.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff0b01dc640 20 client.16379.objecter shutdown clearing up homeless session... 2023-10-30T17:55:10.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.130+0000 7ff0b01dc640 10 client.16379.objecter successfully canceled tick 2023-10-30T17:55:10.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff08c06c850 msgr2=0x7ff08c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7ff08c06c850 0x7ff08c06ed10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8139920 tx=0x7ff098006bf0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 10 monclient: shutdown 2023-10-30T17:55:10.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:10.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0a80b3e20 msgr2=0x7ff0a81447c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 1 --2- 172.21.15.170:0/3950335687 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0a80b3e20 0x7ff0a81447c0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009890 tx=0x7ff0a40046c0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0ae779640 1 -- 172.21.15.170:0/3950335687 reap_dead start 2023-10-30T17:55:10.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 shutdown_connections 2023-10-30T17:55:10.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 >> 172.21.15.170:0/3950335687 conn(0x7ff0a8071770 msgr2=0x7ff0a813bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:10.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 shutdown_connections 2023-10-30T17:55:10.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 1 -- 172.21.15.170:0/3950335687 wait complete. 2023-10-30T17:55:10.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.131+0000 7ff0b01dc640 1 librados: shutdown 2023-10-30T17:55:10.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-30T17:55:10.249 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.248+0000 7f38e2f61640 10 monclient: get_monmap_and_config 2023-10-30T17:55:10.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.248+0000 7f38e2f61640 10 monclient: build_initial_monmap 2023-10-30T17:55:10.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.248+0000 7f38e2f61640 10 monclient: monmap: 2023-10-30T17:55:10.250 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:10.250 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.251 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:10.249390+0000 2023-10-30T17:55:10.251 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:10.249390+0000 2023-10-30T17:55:10.251 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:10.251 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.251 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:10.252 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:10.252 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:10.252 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:10.252 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:10.252 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 1 Processor -- start 2023-10-30T17:55:10.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 1 -- start start 2023-10-30T17:55:10.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 10 monclient: init 2023-10-30T17:55:10.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:10.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-10-30T17:55:10.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8720 0x7f38dc13ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 10 monclient(hunting): picked mon.noname-b con 0x7f38dc0a8720 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:10.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38dc13f4d0 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:10.255 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f38dc0a8b00 0x7f38dc14c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 10 monclient(hunting): picked mon.noname-a con 0x7f38dc0a8b00 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:10.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38dc0a9fd0 con 0x7f38dc13f4d0 2023-10-30T17:55:10.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38dc0aa240 con 0x7f38dc0a8b00 2023-10-30T17:55:10.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc0aa520 con 0x7f38dc0a8720 2023-10-30T17:55:10.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:10.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.249+0000 7f38e2f61640 10 monclient(hunting): authenticate will time out at 2692.572958s 2023-10-30T17:55:10.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.252+0000 7f38dbfff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f38dc13f4d0 0x7f38dc1454e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.170:45966/0 (socket says 172.21.15.170:45966) 2023-10-30T17:55:10.257 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.252+0000 7f38dbfff640 1 -- 172.21.15.170:0/1057859893 learned_addr learned my addr 172.21.15.170:0/1057859893 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:10.258 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.253+0000 7f38e14fe640 1 --2- 172.21.15.170:0/1057859893 >> v2:172.21.15.170:3300/0 conn(0x7f38dc0a8b00 0x7f38dc14c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.259 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.253+0000 7f38e14fe640 10 monclient(hunting): get_auth_request con 0x7f38dc0a8b00 auth_method 0 2023-10-30T17:55:10.259 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.253+0000 7f38e14fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.259 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.253+0000 7f38e14fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.253+0000 7f38e14fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.253+0000 7f38e0cfd640 1 --2- 172.21.15.170:0/1057859893 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8720 0x7f38dc13ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.254+0000 7f38e14fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:10.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.254+0000 7f38e14fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:10.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.254+0000 7f38e14fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.254+0000 7f38e0cfd640 10 monclient(hunting): get_auth_request con 0x7f38dc0a8720 auth_method 0 2023-10-30T17:55:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.254+0000 7f38e0cfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.254+0000 7f38e0cfd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.254+0000 7f38e0cfd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.255+0000 7f38e0cfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:10.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.255+0000 7f38e0cfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:10.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.255+0000 7f38e0cfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:10.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.257+0000 7f38e14fe640 10 monclient(hunting): handle_auth_done global_id 16068 payload 274 2023-10-30T17:55:10.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.257+0000 7f38e14fe640 1 -- 172.21.15.170:0/1057859893 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8720 msgr2=0x7f38dc13ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:10.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.257+0000 7f38e14fe640 1 --2- 172.21.15.170:0/1057859893 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8720 0x7f38dc13ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.257+0000 7f38e14fe640 1 -- 172.21.15.170:0/1057859893 >> v1:172.21.15.177:6789/0 conn(0x7f38dc13f4d0 legacy=0x7f38dc1454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.257+0000 7f38e14fe640 10 monclient: _finish_hunting 0 2023-10-30T17:55:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.257+0000 7f38e14fe640 1 monclient: found mon.noname-a 2023-10-30T17:55:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.257+0000 7f38e14fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.257+0000 7f38e14fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.257+0000 7f38e14fe640 1 -- 172.21.15.170:0/1057859893 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc0b1c80 con 0x7f38dc0a8b00 2023-10-30T17:55:10.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.266+0000 7f38e14fe640 1 --2- 172.21.15.170:0/1057859893 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38dc0a8b00 0x7f38dc14c3b0 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7f38d4009d90 tx=0x7f38d402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7611c9bb3b9e8fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:10.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 1 -- 172.21.15.170:0/1057859893 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d402cc50 con 0x7f38dc0a8b00 2023-10-30T17:55:10.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:10.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 10 monclient: dump: 2023-10-30T17:55:10.275 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.275 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.275 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.276 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.276 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.276 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.276 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.276 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.277 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.277 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 10 monclient: _finish_auth 0 2023-10-30T17:55:10.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:55:10.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:10.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 1 -- 172.21.15.170:0/1057859893 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d402cdb0 con 0x7f38dc0a8b00 2023-10-30T17:55:10.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38e2f61640 5 monclient: authenticate success, global_id 16068 2023-10-30T17:55:10.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38e2f61640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:10.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:10.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38e2f61640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:10.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 1 -- 172.21.15.170:0/1057859893 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4034660 con 0x7f38dc0a8b00 2023-10-30T17:55:10.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38e2f61640 4 set_mon_vals no callback set 2023-10-30T17:55:10.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:10.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.273+0000 7f38db7fe640 10 monclient: dump: 2023-10-30T17:55:10.279 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.280 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.280 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.280 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.280 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.280 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.281 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.281 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.281 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.281 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.274+0000 7f38e2f61640 10 monclient: shutdown 2023-10-30T17:55:10.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.274+0000 7f38e2f61640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:10.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.274+0000 7f38e2f61640 1 -- 172.21.15.170:0/1057859893 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38dc0a8b00 msgr2=0x7f38dc14c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.274+0000 7f38e2f61640 1 --2- 172.21.15.170:0/1057859893 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38dc0a8b00 0x7f38dc14c3b0 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7f38d4009d90 tx=0x7f38d402bea0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.274+0000 7f38e2f61640 1 -- 172.21.15.170:0/1057859893 shutdown_connections 2023-10-30T17:55:10.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.274+0000 7f38e2f61640 1 --2- 172.21.15.170:0/1057859893 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38dc0a8b00 0x7f38dc14c3b0 unknown :-1 s=CLOSED pgs=3381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.274+0000 7f38e2f61640 1 --2- 172.21.15.170:0/1057859893 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8720 0x7f38dc13ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.274+0000 7f38e2f61640 1 -- 172.21.15.170:0/1057859893 >> 172.21.15.170:0/1057859893 conn(0x7f38dc0b7300 msgr2=0x7f38dc0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:10.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.274+0000 7f38e2f61640 1 -- 172.21.15.170:0/1057859893 shutdown_connections 2023-10-30T17:55:10.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.274+0000 7f38e2f61640 1 -- 172.21.15.170:0/1057859893 wait complete. 2023-10-30T17:55:10.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.275+0000 7f38e2f61640 10 monclient: build_initial_monmap 2023-10-30T17:55:10.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.275+0000 7f38e2f61640 10 monclient: monmap: 2023-10-30T17:55:10.287 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:10.287 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:10.287 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:10.287 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:10.287 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:10.288 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.288 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:10.288 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:10.288 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:10.288 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.275+0000 7f38e2f61640 1 librados: starting msgr at 2023-10-30T17:55:10.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.275+0000 7f38e2f61640 1 librados: starting objecter 2023-10-30T17:55:10.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.276+0000 7f38e2f61640 1 Processor -- start 2023-10-30T17:55:10.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.276+0000 7f38e2f61640 1 -- start start 2023-10-30T17:55:10.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.276+0000 7f38e2f61640 1 librados: setting wanted keys 2023-10-30T17:55:10.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.276+0000 7f38e2f61640 1 librados: calling monclient init 2023-10-30T17:55:10.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.276+0000 7f38e2f61640 10 monclient: init 2023-10-30T17:55:10.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:10.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:55:10.290 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38dc0a8720 0x7f38dc144a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 10 monclient(hunting): picked mon.noname-a con 0x7f38dc0a8720 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:10.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8b00 0x7f38dc0b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 10 monclient(hunting): picked mon.noname-c con 0x7f38dc0a8b00 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:10.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38dc13f4d0 0x7f38dc0b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 10 monclient(hunting): picked mon.noname-b con 0x7f38dc13f4d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:10.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc132220 con 0x7f38dc0a8720 2023-10-30T17:55:10.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc1430c0 con 0x7f38dc13f4d0 2023-10-30T17:55:10.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc1442b0 con 0x7f38dc0a8b00 2023-10-30T17:55:10.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:10.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.277+0000 7f38e2f61640 10 monclient(hunting): authenticate will time out at 2692.600354s 2023-10-30T17:55:10.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.278+0000 7f38dbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8b00 0x7f38dc0b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.281+0000 7f38dbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8b00 0x7f38dc0b6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:39166/0 (socket says 172.21.15.170:39166) 2023-10-30T17:55:10.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.281+0000 7f38dbfff640 1 -- 172.21.15.170:0/493869037 learned_addr learned my addr 172.21.15.170:0/493869037 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:10.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.281+0000 7f38dbfff640 10 monclient(hunting): get_auth_request con 0x7f38dc0a8b00 auth_method 0 2023-10-30T17:55:10.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.281+0000 7f38dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.281+0000 7f38dbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.281+0000 7f38dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.281+0000 7f38e14fe640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38dc13f4d0 0x7f38dc0b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.282+0000 7f38e14fe640 10 monclient(hunting): get_auth_request con 0x7f38dc13f4d0 auth_method 0 2023-10-30T17:55:10.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.282+0000 7f38e14fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.282+0000 7f38e14fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.282+0000 7f38e14fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.282+0000 7f38dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:10.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.282+0000 7f38dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:10.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.282+0000 7f38dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:10.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.283+0000 7f38e0cfd640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38dc0a8720 0x7f38dc144a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.285+0000 7f38e14fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:10.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.285+0000 7f38e14fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:10.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.285+0000 7f38e14fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:10.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.285+0000 7f38e0cfd640 10 monclient(hunting): get_auth_request con 0x7f38dc0a8720 auth_method 0 2023-10-30T17:55:10.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.285+0000 7f38e0cfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.285+0000 7f38e0cfd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.285+0000 7f38e0cfd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.286+0000 7f38e0cfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:10.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.286+0000 7f38e0cfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:10.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.286+0000 7f38e0cfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:10.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.287+0000 7f38dbfff640 10 monclient(hunting): handle_auth_done global_id 16385 payload 824 2023-10-30T17:55:10.299 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.287+0000 7f38dbfff640 1 -- 172.21.15.170:0/493869037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38dc13f4d0 msgr2=0x7f38dc0b55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.287+0000 7f38dbfff640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38dc13f4d0 0x7f38dc0b55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.287+0000 7f38dbfff640 1 -- 172.21.15.170:0/493869037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38dc0a8720 msgr2=0x7f38dc144a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.287+0000 7f38dbfff640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38dc0a8720 0x7f38dc144a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.287+0000 7f38dbfff640 10 monclient: _finish_hunting 0 2023-10-30T17:55:10.300 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.287+0000 7f38dbfff640 1 monclient: found mon.noname-c 2023-10-30T17:55:10.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.287+0000 7f38dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:10.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.287+0000 7f38dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.170:3301/0 2023-10-30T17:55:10.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.287+0000 7f38dbfff640 1 -- 172.21.15.170:0/493869037 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d4009a70 con 0x7f38dc0a8b00 2023-10-30T17:55:10.301 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:54.260 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:10.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.293+0000 7f38dbfff640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8b00 0x7f38dc0b6480 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f38c800c3c0 tx=0x7f38c800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:10.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.293+0000 7f38d97fa640 10 client.?.objecter ms_handle_connect 0x7f38dc0a8b00 2023-10-30T17:55:10.302 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.293+0000 7f38d97fa640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:10.303 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:54.261 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:10.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 1 -- 172.21.15.170:0/493869037 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c8005ce0 con 0x7f38dc0a8b00 2023-10-30T17:55:10.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:10.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 10 monclient: dump: 2023-10-30T17:55:10.305 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.305 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.305 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.305 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.305 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.306 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.306 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.306 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.306 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.306 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 10 monclient: _finish_auth 0 2023-10-30T17:55:10.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 10 monclient: _check_auth_tickets 2023-10-30T17:55:10.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:10.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 1 -- 172.21.15.170:0/493869037 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c8005e40 con 0x7f38dc0a8b00 2023-10-30T17:55:10.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:10.307 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 1 -- 172.21.15.170:0/493869037 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c80072d0 con 0x7f38dc0a8b00 2023-10-30T17:55:10.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38e2f61640 5 monclient: authenticate success, global_id 16385 2023-10-30T17:55:10.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T17:55:10.308 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38d97fa640 10 monclient: dump: 2023-10-30T17:55:10.308 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.309 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.309 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.309 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.309 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.309 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.310 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.310 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.310 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.310 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38e2f61640 10 monclient: _renew_subs 2023-10-30T17:55:10.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38e2f61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:55:10.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38daffd640 4 set_mon_vals no callback set 2023-10-30T17:55:10.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38e2f61640 1 -- 172.21.15.170:0/493869037 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38dc142de0 con 0x7f38dc0a8b00 2023-10-30T17:55:10.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38e2f61640 10 client.16385.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:10.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38e2f61640 10 monclient: _renew_subs 2023-10-30T17:55:10.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38e2f61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:55:10.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38e2f61640 1 -- 172.21.15.170:0/493869037 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38dc13b370 con 0x7f38dc0a8b00 2023-10-30T17:55:10.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.301+0000 7f38e2f61640 1 librados: init done 2023-10-30T17:55:10.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.302+0000 7f38e2f61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:10.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.302+0000 7f38e2f61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:10.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.302+0000 7f38e2f61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.170:3301/0 2023-10-30T17:55:10.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.302+0000 7f38e2f61640 1 -- 172.21.15.170:0/493869037 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38dc13b090 con 0x7f38dc0a8b00 2023-10-30T17:55:10.313 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:54.270 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:10.484 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:54.423 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:10.485 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:54.424 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:10.493 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:54.432 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:10.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.546+0000 7f38d97fa640 1 -- 172.21.15.170:0/493869037 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f38c8007430 con 0x7f38dc0a8b00 2023-10-30T17:55:10.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.546+0000 7f38d97fa640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f38c006c880 0x7f38c006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.547+0000 7f38e0cfd640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f38c006c880 0x7f38c006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.547+0000 7f38e0cfd640 10 monclient: get_auth_request con 0x7f38c006c880 auth_method 0 2023-10-30T17:55:10.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.547+0000 7f38e0cfd640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f38c006c880 0x7f38c006ed40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f38d0009b40 tx=0x7f38d0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:10.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.547+0000 7f38d97fa640 10 client.16385.objecter ms_handle_connect 0x7f38c006c880 2023-10-30T17:55:10.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.561+0000 7f38d97fa640 1 -- 172.21.15.170:0/493869037 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(914..914 src has 703..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f38c8057b80 con 0x7f38dc0a8b00 2023-10-30T17:55:10.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.561+0000 7f38d97fa640 10 client.16385.objecter ms_dispatch 0x7f38dc000b90 osd_map(914..914 src has 703..914) v4 2023-10-30T17:55:10.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.561+0000 7f38d97fa640 3 client.16385.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:10.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.561+0000 7f38d97fa640 3 client.16385.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:10.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.561+0000 7f38d97fa640 20 client.16385.objecter dump_active .. 0 homeless 2023-10-30T17:55:10.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.561+0000 7f38d97fa640 1 -- 172.21.15.170:0/493869037 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f38c8053ec0 con 0x7f38dc0a8b00 2023-10-30T17:55:10.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.561+0000 7f38d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:10.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.561+0000 7f38d97fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:10.650 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:54.616 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:10.651 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:54.617 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:10.660 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:54.626 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:10.743 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.742+0000 7f38e2f61640 1 -- 172.21.15.170:0/493869037 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f38dc13b090 con 0x7f38c006c880 2023-10-30T17:55:10.744 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.742+0000 7f38d97fa640 1 -- 172.21.15.170:0/493869037 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f38dc13b090 con 0x7f38c006c880 2023-10-30T17:55:10.744 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 7 to deep-scrub 2023-10-30T17:55:10.747 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 10 librados: watch_flush enter 2023-10-30T17:55:10.747 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 10 librados: watch_flush exit 2023-10-30T17:55:10.747 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 20 client.16385.objecter shutdown clearing up homeless session... 2023-10-30T17:55:10.747 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 10 client.16385.objecter successfully canceled tick 2023-10-30T17:55:10.747 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 -- 172.21.15.170:0/493869037 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f38c006c880 msgr2=0x7f38c006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f38c006c880 0x7f38c006ed40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f38d0009b40 tx=0x7f38d0009190 comp rx=0 tx=0).stop 2023-10-30T17:55:10.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 10 monclient: shutdown 2023-10-30T17:55:10.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:10.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 -- 172.21.15.170:0/493869037 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8b00 msgr2=0x7f38dc0b6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.748 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8b00 0x7f38dc0b6480 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f38c800c3c0 tx=0x7f38c800c880 comp rx=0 tx=0).stop 2023-10-30T17:55:10.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 -- 172.21.15.170:0/493869037 shutdown_connections 2023-10-30T17:55:10.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f38c006c880 0x7f38c006ed40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38dc13f4d0 0x7f38dc0b55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38dc0a8b00 0x7f38dc0b6480 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 --2- 172.21.15.170:0/493869037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38dc0a8720 0x7f38dc144a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 -- 172.21.15.170:0/493869037 >> 172.21.15.170:0/493869037 conn(0x7f38dc0b7300 msgr2=0x7f38dc145120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:10.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 -- 172.21.15.170:0/493869037 shutdown_connections 2023-10-30T17:55:10.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 -- 172.21.15.170:0/493869037 wait complete. 2023-10-30T17:55:10.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.745+0000 7f38e2f61640 1 librados: shutdown 2023-10-30T17:55:10.757 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T17:55:10.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.857+0000 7f2d1f2e0640 10 monclient: get_monmap_and_config 2023-10-30T17:55:10.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.857+0000 7f2d1f2e0640 10 monclient: build_initial_monmap 2023-10-30T17:55:10.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.857+0000 7f2d1f2e0640 10 monclient: monmap: 2023-10-30T17:55:10.859 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:10.859 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.859 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:10.857968+0000 2023-10-30T17:55:10.859 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:10.857968+0000 2023-10-30T17:55:10.859 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:10.861 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.861 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:10.861 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:10.862 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:10.862 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:10.862 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:10.862 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.864 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.857+0000 7f2d1f2e0640 1 Processor -- start 2023-10-30T17:55:10.864 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.857+0000 7f2d1f2e0640 1 -- start start 2023-10-30T17:55:10.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.857+0000 7f2d1f2e0640 10 monclient: init 2023-10-30T17:55:10.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:10.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T17:55:10.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2d180b7c90 0x7f2d180a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 10 monclient(hunting): picked mon.noname-c con 0x7f2d180b7c90 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:10.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d1813eb60 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:10.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2d18144f50 0x7f2d18145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 10 monclient(hunting): picked mon.noname-a con 0x7f2d18144f50 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:10.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d180aa740 con 0x7f2d1813eb60 2023-10-30T17:55:10.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d180a9500 con 0x7f2d18144f50 2023-10-30T17:55:10.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1813f6c0 con 0x7f2d180b7c90 2023-10-30T17:55:10.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:10.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.858+0000 7f2d1f2e0640 10 monclient(hunting): authenticate will time out at 2693.181482s 2023-10-30T17:55:10.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.859+0000 7f2d1c87b640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f2d1813eb60 0x7f2d1813efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:37130/0 (socket says 172.21.15.170:37130) 2023-10-30T17:55:10.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.859+0000 7f2d1c87b640 1 -- 172.21.15.170:0/3331119540 learned_addr learned my addr 172.21.15.170:0/3331119540 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:10.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.861+0000 7f2d1d87d640 1 --2- 172.21.15.170:0/3331119540 >> v2:172.21.15.170:3300/0 conn(0x7f2d18144f50 0x7f2d18145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.861+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 <== mon.0 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926553466 0 0) 0x7f2d180aa740 con 0x7f2d1813eb60 2023-10-30T17:55:10.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.861+0000 7f2d0ffff640 10 monclient(hunting): _negotiate 2023-10-30T17:55:10.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.861+0000 7f2d0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.861+0000 7f2d0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.861+0000 7f2d0ffff640 10 monclient(hunting): my global_id is 16074 2023-10-30T17:55:10.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.861+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d04003520 con 0x7f2d1813eb60 2023-10-30T17:55:10.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d1d07c640 1 --2- 172.21.15.170:0/3331119540 >> v2:172.21.15.177:3300/0 conn(0x7f2d180b7c90 0x7f2d180a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d1d87d640 10 monclient(hunting): get_auth_request con 0x7f2d18144f50 auth_method 0 2023-10-30T17:55:10.872 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d1d87d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d1d87d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d1d87d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 <== mon.0 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1725699399 0 0) 0x7f2d04003520 con 0x7f2d1813eb60 2023-10-30T17:55:10.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 >> v2:172.21.15.177:3300/0 conn(0x7f2d180b7c90 msgr2=0x7f2d180a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:10.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 --2- 172.21.15.170:0/3331119540 >> v2:172.21.15.177:3300/0 conn(0x7f2d180b7c90 0x7f2d180a8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 >> v2:172.21.15.170:3300/0 conn(0x7f2d18144f50 msgr2=0x7f2d18145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:10.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 --2- 172.21.15.170:0/3331119540 >> v2:172.21.15.170:3300/0 conn(0x7f2d18144f50 0x7f2d18145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient: _finish_hunting 0 2023-10-30T17:55:10.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 monclient: found mon.noname-a-legacy 2023-10-30T17:55:10.874 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:10.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.170:6789/0 2023-10-30T17:55:10.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d180aa9b0 con 0x7f2d1813eb60 2023-10-30T17:55:10.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient: _finish_auth 0 2023-10-30T17:55:10.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient: _check_auth_tickets 2023-10-30T17:55:10.875 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:10.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 <== mon.0 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2465063691 0 0) 0x7f2d08001670 con 0x7f2d1813eb60 2023-10-30T17:55:10.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d1f2e0640 5 monclient: authenticate success, global_id 16074 2023-10-30T17:55:10.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d1f2e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:10.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.876 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T17:55:10.877 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient: dump: 2023-10-30T17:55:10.877 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.877 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.877 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.877 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.877 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.878 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.878 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.878 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.878 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.878 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] but i'm connected to v1:172.21.15.170:6789/0, reconnecting 2023-10-30T17:55:10.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:10.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 >> v1:172.21.15.170:6789/0 conn(0x7f2d1813eb60 legacy=0x7f2d1813efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T17:55:10.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 --2- 172.21.15.170:0/3331119540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d040039c0 0x7f2d04003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient(hunting): picked mon.a con 0x7f2d040039c0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:10.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 --2- 172.21.15.170:0/3331119540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d040047c0 0x7f2d04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient(hunting): picked mon.c con 0x7f2d040047c0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:10.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 --2- 172.21.15.170:0/3331119540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d040051d0 0x7f2d0400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient(hunting): picked mon.b con 0x7f2d040051d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:10.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d180a9500 con 0x7f2d040039c0 2023-10-30T17:55:10.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1813f6c0 con 0x7f2d040051d0 2023-10-30T17:55:10.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0400c1a0 con 0x7f2d040047c0 2023-10-30T17:55:10.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:10.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d0ffff640 10 monclient(hunting): _finish_auth 0 2023-10-30T17:55:10.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.865+0000 7f2d1f2e0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:10.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.869+0000 7f2d1d07c640 1 --2- 172.21.15.170:0/3331119540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d040039c0 0x7f2d04003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.869+0000 7f2d1c87b640 1 --2- 172.21.15.170:0/3331119540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d040047c0 0x7f2d04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.869+0000 7f2d1d07c640 10 monclient(hunting): get_auth_request con 0x7f2d040039c0 auth_method 0 2023-10-30T17:55:10.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.869+0000 7f2d1d07c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.869+0000 7f2d1d07c640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.869+0000 7f2d1d07c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T17:55:10.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.870+0000 7f2d1d07c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:10.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.870+0000 7f2d1d07c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:10.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.870+0000 7f2d1d07c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T17:55:10.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 10 monclient(hunting): handle_auth_done global_id 16074 payload 293 2023-10-30T17:55:10.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 1 -- 172.21.15.170:0/3331119540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d040047c0 msgr2=0x7f2d04004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:10.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 1 --2- 172.21.15.170:0/3331119540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d040047c0 0x7f2d04004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 1 -- 172.21.15.170:0/3331119540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d040051d0 msgr2=0x7f2d0400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:10.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 1 --2- 172.21.15.170:0/3331119540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d040051d0 0x7f2d0400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 10 monclient: _finish_hunting 0 2023-10-30T17:55:10.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 1 monclient: found mon.a 2023-10-30T17:55:10.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:10.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d87d640 1 -- 172.21.15.170:0/3331119540 reap_dead start 2023-10-30T17:55:10.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:10.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 1 -- 172.21.15.170:0/3331119540 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d04003520 con 0x7f2d040039c0 2023-10-30T17:55:10.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 10 monclient: _finish_auth 0 2023-10-30T17:55:10.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 10 monclient: _check_auth_tickets 2023-10-30T17:55:10.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.872+0000 7f2d1d07c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:10.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.874+0000 7f2d1d07c640 1 --2- 172.21.15.170:0/3331119540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d040039c0 0x7f2d04003de0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7f2d10007f60 tx=0x7f2d040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6e41682f5ae2aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:10.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d10033600 con 0x7f2d040039c0 2023-10-30T17:55:10.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:10.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d0ffff640 10 monclient: dump: 2023-10-30T17:55:10.892 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.892 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.892 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.892 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.892 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.893 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.893 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.893 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.893 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.893 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1002f050 con 0x7f2d040039c0 2023-10-30T17:55:10.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d1f2e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:10.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d0ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:10.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d1f2e0640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:10.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d0ffff640 1 -- 172.21.15.170:0/3331119540 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d100339e0 con 0x7f2d040039c0 2023-10-30T17:55:10.895 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.895 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d1f2e0640 4 set_mon_vals no callback set 2023-10-30T17:55:10.895 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:10.895 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.878+0000 7f2d0ffff640 10 monclient: dump: 2023-10-30T17:55:10.895 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.896 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.896 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.896 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.896 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.896 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.897 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.897 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.897 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.897 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.879+0000 7f2d1f2e0640 10 monclient: shutdown 2023-10-30T17:55:10.898 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.879+0000 7f2d1f2e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:10.898 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.879+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/3331119540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d040039c0 msgr2=0x7f2d04003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.898 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.879+0000 7f2d1f2e0640 1 --2- 172.21.15.170:0/3331119540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d040039c0 0x7f2d04003de0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7f2d10007f60 tx=0x7f2d040057f0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.898 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.879+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/3331119540 shutdown_connections 2023-10-30T17:55:10.898 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.879+0000 7f2d1f2e0640 1 --2- 172.21.15.170:0/3331119540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d040039c0 0x7f2d04003de0 unknown :-1 s=CLOSED pgs=3383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.879+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/3331119540 >> 172.21.15.170:0/3331119540 conn(0x7f2d18071da0 msgr2=0x7f2d180b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:10.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.879+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/3331119540 shutdown_connections 2023-10-30T17:55:10.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.879+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/3331119540 wait complete. 2023-10-30T17:55:10.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.880+0000 7f2d1f2e0640 10 monclient: build_initial_monmap 2023-10-30T17:55:10.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.880+0000 7f2d1f2e0640 10 monclient: monmap: 2023-10-30T17:55:10.900 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:10.900 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:10.900 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:10.900 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:10.901 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:10.901 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.901 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:10.901 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:10.901 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:10.902 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.880+0000 7f2d1f2e0640 1 librados: starting msgr at 2023-10-30T17:55:10.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.880+0000 7f2d1f2e0640 1 librados: starting objecter 2023-10-30T17:55:10.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.880+0000 7f2d1f2e0640 1 Processor -- start 2023-10-30T17:55:10.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.880+0000 7f2d1f2e0640 1 -- start start 2023-10-30T17:55:10.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.880+0000 7f2d1f2e0640 1 librados: setting wanted keys 2023-10-30T17:55:10.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.880+0000 7f2d1f2e0640 1 librados: calling monclient init 2023-10-30T17:55:10.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.880+0000 7f2d1f2e0640 10 monclient: init 2023-10-30T17:55:10.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:10.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T17:55:10.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d040039c0 0x7f2d18142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 10 monclient(hunting): picked mon.noname-c con 0x7f2d040039c0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:10.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d18143290 0x7f2d180b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 10 monclient(hunting): picked mon.noname-b con 0x7f2d18143290 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:10.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18144a30 0x7f2d180b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:10.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 10 monclient(hunting): picked mon.noname-a con 0x7f2d18144a30 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:10.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18142150 con 0x7f2d18144a30 2023-10-30T17:55:10.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18141370 con 0x7f2d18143290 2023-10-30T17:55:10.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:10.907 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d181d9d20 con 0x7f2d040039c0 2023-10-30T17:55:10.907 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:10.907 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1f2e0640 10 monclient(hunting): authenticate will time out at 2693.204256s 2023-10-30T17:55:10.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1d87d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18144a30 0x7f2d180b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1d87d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18144a30 0x7f2d180b0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:41968/0 (socket says 172.21.15.170:41968) 2023-10-30T17:55:10.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1d87d640 1 -- 172.21.15.170:0/4252473611 learned_addr learned my addr 172.21.15.170:0/4252473611 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:10.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1d87d640 10 monclient(hunting): get_auth_request con 0x7f2d18144a30 auth_method 0 2023-10-30T17:55:10.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1d87d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.909 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1d87d640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.909 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.881+0000 7f2d1d87d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.909 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.882+0000 7f2d1d87d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:10.909 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.882+0000 7f2d1d87d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:10.909 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.882+0000 7f2d1d87d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:10.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.883+0000 7f2d1c87b640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d18143290 0x7f2d180b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.885+0000 7f2d1d07c640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d040039c0 0x7f2d18142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:10.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.885+0000 7f2d1c87b640 10 monclient(hunting): get_auth_request con 0x7f2d18143290 auth_method 0 2023-10-30T17:55:10.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.885+0000 7f2d1c87b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:10.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.885+0000 7f2d1c87b640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:10.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.885+0000 7f2d1c87b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:10.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.886+0000 7f2d1d87d640 10 monclient(hunting): handle_auth_done global_id 16080 payload 824 2023-10-30T17:55:10.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.886+0000 7f2d1d87d640 1 -- 172.21.15.170:0/4252473611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d040039c0 msgr2=0x7f2d18142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.886+0000 7f2d1d87d640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d040039c0 0x7f2d18142d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.886+0000 7f2d1d87d640 1 -- 172.21.15.170:0/4252473611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d18143290 msgr2=0x7f2d180b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:10.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.886+0000 7f2d1d87d640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d18143290 0x7f2d180b01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:10.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.886+0000 7f2d1d87d640 10 monclient: _finish_hunting 0 2023-10-30T17:55:10.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.886+0000 7f2d1d87d640 1 monclient: found mon.noname-a 2023-10-30T17:55:10.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.886+0000 7f2d1d87d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:10.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.886+0000 7f2d1d87d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:10.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.886+0000 7f2d1d87d640 1 -- 172.21.15.170:0/4252473611 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d181d9ee0 con 0x7f2d18144a30 2023-10-30T17:55:10.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.889+0000 7f2d1d87d640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18144a30 0x7f2d180b0720 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7f2d180b2d10 tx=0x7f2d14007820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:10.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.889+0000 7f2d0dffb640 10 client.?.objecter ms_handle_connect 0x7f2d18144a30 2023-10-30T17:55:10.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.889+0000 7f2d0dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:10.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 1 -- 172.21.15.170:0/4252473611 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14004240 con 0x7f2d18144a30 2023-10-30T17:55:10.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:10.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 10 monclient: dump: 2023-10-30T17:55:10.916 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.916 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.916 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.917 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.917 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.917 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.917 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.917 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.918 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.918 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.918 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 10 monclient: _finish_auth 0 2023-10-30T17:55:10.918 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 10 monclient: _check_auth_tickets 2023-10-30T17:55:10.918 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:10.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d1f2e0640 5 monclient: authenticate success, global_id 16080 2023-10-30T17:55:10.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 1 -- 172.21.15.170:0/4252473611 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d140043a0 con 0x7f2d18144a30 2023-10-30T17:55:10.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:10.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 1 -- 172.21.15.170:0/4252473611 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14003850 con 0x7f2d18144a30 2023-10-30T17:55:10.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:10.920 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:10.920 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0dffb640 10 monclient: dump: 2023-10-30T17:55:10.920 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:10.920 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:10.920 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.921 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:10.922 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:10.922 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:10.922 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:10.922 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:10.923 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:10.923 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:10.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d0f7fe640 4 set_mon_vals no callback set 2023-10-30T17:55:10.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d1f2e0640 10 monclient: _renew_subs 2023-10-30T17:55:10.923 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d1f2e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:10.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/4252473611 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d181da130 con 0x7f2d18144a30 2023-10-30T17:55:10.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d1f2e0640 10 client.16080.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:10.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d1f2e0640 10 monclient: _renew_subs 2023-10-30T17:55:10.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d1f2e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:10.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/4252473611 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d181da5c0 con 0x7f2d18144a30 2023-10-30T17:55:10.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.893+0000 7f2d1f2e0640 1 librados: init done 2023-10-30T17:55:10.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.894+0000 7f2d1f2e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:10.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.894+0000 7f2d1f2e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:10.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.894+0000 7f2d1f2e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:10.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:10.894+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/4252473611 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d18141fb0 con 0x7f2d18144a30 2023-10-30T17:55:11.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.146+0000 7f2d0dffb640 1 -- 172.21.15.170:0/4252473611 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2d140039b0 con 0x7f2d18144a30 2023-10-30T17:55:11.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.147+0000 7f2d0dffb640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2cfc06c800 0x7f2cfc06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:11.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.147+0000 7f2d1d07c640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2cfc06c800 0x7f2cfc06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:11.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.147+0000 7f2d1d07c640 10 monclient: get_auth_request con 0x7f2cfc06c800 auth_method 0 2023-10-30T17:55:11.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.148+0000 7f2d1d07c640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2cfc06c800 0x7f2cfc06ecc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2d10036000 tx=0x7f2d10035040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:11.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.148+0000 7f2d0dffb640 10 client.16080.objecter ms_handle_connect 0x7f2cfc06c800 2023-10-30T17:55:11.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.165+0000 7f2d0dffb640 1 -- 172.21.15.170:0/4252473611 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(914..914 src has 703..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f2d140580a0 con 0x7f2d18144a30 2023-10-30T17:55:11.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.165+0000 7f2d0dffb640 10 client.16080.objecter ms_dispatch 0x7f2d18000b90 osd_map(914..914 src has 703..914) v4 2023-10-30T17:55:11.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.165+0000 7f2d0dffb640 3 client.16080.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:11.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.165+0000 7f2d0dffb640 3 client.16080.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:11.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.165+0000 7f2d0dffb640 20 client.16080.objecter dump_active .. 0 homeless 2023-10-30T17:55:11.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.165+0000 7f2d0dffb640 1 -- 172.21.15.170:0/4252473611 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2d14053ce0 con 0x7f2d18144a30 2023-10-30T17:55:11.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.165+0000 7f2d0dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:11.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.165+0000 7f2d0dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:11.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.362+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/4252473611 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d180a9900 con 0x7f2cfc06c800 2023-10-30T17:55:11.368 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.367+0000 7f2d0dffb640 1 -- 172.21.15.170:0/4252473611 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+150925 (secure 0 0 0) 0x7f2d100337b0 con 0x7f2cfc06c800 2023-10-30T17:55:11.370 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:55:11.371 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-10-30T17:55:11.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.369+0000 7f2d1f2e0640 10 librados: watch_flush enter 2023-10-30T17:55:11.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 10 librados: watch_flush exit 2023-10-30T17:55:11.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 20 client.16080.objecter shutdown clearing up homeless session... 2023-10-30T17:55:11.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 10 client.16080.objecter successfully canceled tick 2023-10-30T17:55:11.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/4252473611 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2cfc06c800 msgr2=0x7f2cfc06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:11.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2cfc06c800 0x7f2cfc06ecc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2d10036000 tx=0x7f2d10035040 comp rx=0 tx=0).stop 2023-10-30T17:55:11.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 10 monclient: shutdown 2023-10-30T17:55:11.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:11.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/4252473611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18144a30 msgr2=0x7f2d180b0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:11.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18144a30 0x7f2d180b0720 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7f2d180b2d10 tx=0x7f2d14007820 comp rx=0 tx=0).stop 2023-10-30T17:55:11.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/4252473611 shutdown_connections 2023-10-30T17:55:11.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f2cfc06c800 0x7f2cfc06ecc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:11.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d18143290 0x7f2d180b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:11.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18144a30 0x7f2d180b0720 unknown :-1 s=CLOSED pgs=3384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:11.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 --2- 172.21.15.170:0/4252473611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d040039c0 0x7f2d18142d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:11.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/4252473611 >> 172.21.15.170:0/4252473611 conn(0x7f2d18071da0 msgr2=0x7f2d1813dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:11.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/4252473611 shutdown_connections 2023-10-30T17:55:11.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 -- 172.21.15.170:0/4252473611 wait complete. 2023-10-30T17:55:11.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:11.370+0000 7f2d1f2e0640 1 librados: shutdown 2023-10-30T17:55:11.382 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2023-10-30T17:55:11.327935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":568588,"kb_used_data":60984,"kb_used_omap":12,"kb_used_meta":507507,"kb_avail":754406132,"statfs":{"total":773094113280,"available":772511879168,"internally_reserved":0,"allocated":62447616,"data_stored":34691458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":519687736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002161"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":861,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.035987+0000","last_change":"2023-10-30T17:33:37.648571+0000","last_active":"2023-10-30T17:55:01.035987+0000","last_peered":"2023-10-30T17:55:01.035987+0000","last_clean":"2023-10-30T17:55:01.035987+0000","last_became_active":"2023-10-30T17:33:37.643213+0000","last_became_peered":"2023-10-30T17:33:37.643213+0000","last_unstale":"2023-10-30T17:55:01.035987+0000","last_undegraded":"2023-10-30T17:55:01.035987+0000","last_fullsized":"2023-10-30T17:55:01.035987+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:43:30.718487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":863,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:11.163105+0000","last_change":"2023-10-30T17:55:11.164191+0000","last_active":"2023-10-30T17:55:11.163105+0000","last_peered":"2023-10-30T17:55:11.163105+0000","last_clean":"2023-10-30T17:55:11.163105+0000","last_became_active":"2023-10-30T17:33:37.827806+0000","last_became_peered":"2023-10-30T17:33:37.827806+0000","last_unstale":"2023-10-30T17:55:11.163105+0000","last_undegraded":"2023-10-30T17:55:11.163105+0000","last_fullsized":"2023-10-30T17:55:11.163105+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:11.161754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:11.161754+0000","last_clean_scrub_stamp":"2023-10-30T17:55:11.161754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:38.406882+0000","scrub_duration":0.050985619000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":856,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.945512+0000","last_change":"2023-10-30T17:33:40.825119+0000","last_active":"2023-10-30T17:55:04.945512+0000","last_peered":"2023-10-30T17:55:04.945512+0000","last_clean":"2023-10-30T17:55:04.945512+0000","last_became_active":"2023-10-30T17:33:40.511287+0000","last_became_peered":"2023-10-30T17:33:40.511287+0000","last_unstale":"2023-10-30T17:55:04.945512+0000","last_undegraded":"2023-10-30T17:55:04.945512+0000","last_fullsized":"2023-10-30T17:55:04.945512+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":855,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.934533+0000","last_change":"2023-10-30T17:33:38.590694+0000","last_active":"2023-10-30T17:55:04.934533+0000","last_peered":"2023-10-30T17:55:04.934533+0000","last_clean":"2023-10-30T17:55:04.934533+0000","last_became_active":"2023-10-30T17:33:38.581457+0000","last_became_peered":"2023-10-30T17:33:38.581457+0000","last_unstale":"2023-10-30T17:55:04.934533+0000","last_undegraded":"2023-10-30T17:55:04.934533+0000","last_fullsized":"2023-10-30T17:55:04.934533+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":861,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.047783+0000","last_change":"2023-10-30T17:33:46.635360+0000","last_active":"2023-10-30T17:55:01.047783+0000","last_peered":"2023-10-30T17:55:01.047783+0000","last_clean":"2023-10-30T17:55:01.047783+0000","last_became_active":"2023-10-30T17:33:46.629442+0000","last_became_peered":"2023-10-30T17:33:46.629442+0000","last_unstale":"2023-10-30T17:55:01.047783+0000","last_undegraded":"2023-10-30T17:55:01.047783+0000","last_fullsized":"2023-10-30T17:55:01.047783+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:46:06.082348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":856,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.942001+0000","last_change":"2023-10-30T17:33:39.141659+0000","last_active":"2023-10-30T17:55:04.942001+0000","last_peered":"2023-10-30T17:55:04.942001+0000","last_clean":"2023-10-30T17:55:04.942001+0000","last_became_active":"2023-10-30T17:33:39.135750+0000","last_became_peered":"2023-10-30T17:33:39.135750+0000","last_unstale":"2023-10-30T17:55:04.942001+0000","last_undegraded":"2023-10-30T17:55:04.942001+0000","last_fullsized":"2023-10-30T17:55:04.942001+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":857,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.951963+0000","last_change":"2023-10-30T17:33:51.011431+0000","last_active":"2023-10-30T17:55:04.951963+0000","last_peered":"2023-10-30T17:55:04.951963+0000","last_clean":"2023-10-30T17:55:04.951963+0000","last_became_active":"2023-10-30T17:33:51.001524+0000","last_became_peered":"2023-10-30T17:33:51.001524+0000","last_unstale":"2023-10-30T17:55:04.951963+0000","last_undegraded":"2023-10-30T17:55:04.951963+0000","last_fullsized":"2023-10-30T17:55:04.951963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.297061+0000","last_change":"2023-10-30T17:33:58.879816+0000","last_active":"2023-10-30T17:55:04.297061+0000","last_peered":"2023-10-30T17:55:04.297061+0000","last_clean":"2023-10-30T17:55:04.297061+0000","last_became_active":"2023-10-30T17:33:58.874661+0000","last_became_peered":"2023-10-30T17:33:58.874661+0000","last_unstale":"2023-10-30T17:55:04.297061+0000","last_undegraded":"2023-10-30T17:55:04.297061+0000","last_fullsized":"2023-10-30T17:55:04.297061+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":852,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.435705+0000","last_change":"2023-10-30T17:33:39.104029+0000","last_active":"2023-10-30T17:55:04.435705+0000","last_peered":"2023-10-30T17:55:04.435705+0000","last_clean":"2023-10-30T17:55:04.435705+0000","last_became_active":"2023-10-30T17:33:39.086390+0000","last_became_peered":"2023-10-30T17:33:39.086390+0000","last_unstale":"2023-10-30T17:55:04.435705+0000","last_undegraded":"2023-10-30T17:55:04.435705+0000","last_fullsized":"2023-10-30T17:55:04.435705+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":857,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.937385+0000","last_change":"2023-10-30T17:33:40.038786+0000","last_active":"2023-10-30T17:55:04.937385+0000","last_peered":"2023-10-30T17:55:04.937385+0000","last_clean":"2023-10-30T17:55:04.937385+0000","last_became_active":"2023-10-30T17:33:39.965244+0000","last_became_peered":"2023-10-30T17:33:39.965244+0000","last_unstale":"2023-10-30T17:55:04.937385+0000","last_undegraded":"2023-10-30T17:55:04.937385+0000","last_fullsized":"2023-10-30T17:55:04.937385+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":862,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.039576+0000","last_change":"2023-10-30T17:33:46.567515+0000","last_active":"2023-10-30T17:55:01.039576+0000","last_peered":"2023-10-30T17:55:01.039576+0000","last_clean":"2023-10-30T17:55:01.039576+0000","last_became_active":"2023-10-30T17:33:46.562080+0000","last_became_peered":"2023-10-30T17:33:46.562080+0000","last_unstale":"2023-10-30T17:55:01.039576+0000","last_undegraded":"2023-10-30T17:55:01.039576+0000","last_fullsized":"2023-10-30T17:55:01.039576+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:42:52.518433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.280658+0000","last_change":"2023-10-30T17:33:40.831046+0000","last_active":"2023-10-30T17:55:04.280658+0000","last_peered":"2023-10-30T17:55:04.280658+0000","last_clean":"2023-10-30T17:55:04.280658+0000","last_became_active":"2023-10-30T17:33:40.798989+0000","last_became_peered":"2023-10-30T17:33:40.798989+0000","last_unstale":"2023-10-30T17:55:04.280658+0000","last_undegraded":"2023-10-30T17:55:04.280658+0000","last_fullsized":"2023-10-30T17:55:04.280658+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":856,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.455890+0000","last_change":"2023-10-30T17:32:35.372482+0000","last_active":"2023-10-30T17:55:04.455890+0000","last_peered":"2023-10-30T17:55:04.455890+0000","last_clean":"2023-10-30T17:55:04.455890+0000","last_became_active":"2023-10-30T17:32:35.367487+0000","last_became_peered":"2023-10-30T17:32:35.367487+0000","last_unstale":"2023-10-30T17:55:04.455890+0000","last_undegraded":"2023-10-30T17:55:04.455890+0000","last_fullsized":"2023-10-30T17:55:04.455890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":863,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.029053+0000","last_change":"2023-10-30T17:33:38.291772+0000","last_active":"2023-10-30T17:55:01.029053+0000","last_peered":"2023-10-30T17:55:01.029053+0000","last_clean":"2023-10-30T17:55:01.029053+0000","last_became_active":"2023-10-30T17:33:38.284317+0000","last_became_peered":"2023-10-30T17:33:38.284317+0000","last_unstale":"2023-10-30T17:55:01.029053+0000","last_undegraded":"2023-10-30T17:55:01.029053+0000","last_fullsized":"2023-10-30T17:55:01.029053+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:02.583403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":868,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.055660+0000","last_change":"2023-10-30T17:32:34.589393+0000","last_active":"2023-10-30T17:55:01.055660+0000","last_peered":"2023-10-30T17:55:01.055660+0000","last_clean":"2023-10-30T17:55:01.055660+0000","last_became_active":"2023-10-30T17:32:34.557021+0000","last_became_peered":"2023-10-30T17:32:34.557021+0000","last_unstale":"2023-10-30T17:55:01.055660+0000","last_undegraded":"2023-10-30T17:55:01.055660+0000","last_fullsized":"2023-10-30T17:55:01.055660+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:05.352859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"47'7","reported_seq":866,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.434107+0000","last_change":"2023-10-30T17:33:39.222547+0000","last_active":"2023-10-30T17:55:04.434107+0000","last_peered":"2023-10-30T17:55:04.434107+0000","last_clean":"2023-10-30T17:55:04.434107+0000","last_became_active":"2023-10-30T17:33:39.211129+0000","last_became_peered":"2023-10-30T17:33:39.211129+0000","last_unstale":"2023-10-30T17:55:04.434107+0000","last_undegraded":"2023-10-30T17:55:04.434107+0000","last_fullsized":"2023-10-30T17:55:04.434107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":857,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.449598+0000","last_change":"2023-10-30T17:32:35.362652+0000","last_active":"2023-10-30T17:55:04.449598+0000","last_peered":"2023-10-30T17:55:04.449598+0000","last_clean":"2023-10-30T17:55:04.449598+0000","last_became_active":"2023-10-30T17:32:35.357489+0000","last_became_peered":"2023-10-30T17:32:35.357489+0000","last_unstale":"2023-10-30T17:55:04.449598+0000","last_undegraded":"2023-10-30T17:55:04.449598+0000","last_fullsized":"2023-10-30T17:55:04.449598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":869,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:06.175371+0000","last_change":"2023-10-30T17:55:06.176783+0000","last_active":"2023-10-30T17:55:06.175371+0000","last_peered":"2023-10-30T17:55:06.175371+0000","last_clean":"2023-10-30T17:55:06.175371+0000","last_became_active":"2023-10-30T17:33:40.584323+0000","last_became_peered":"2023-10-30T17:33:40.584323+0000","last_unstale":"2023-10-30T17:55:06.175371+0000","last_undegraded":"2023-10-30T17:55:06.175371+0000","last_fullsized":"2023-10-30T17:55:06.175371+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:06.173670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:06.173670+0000","last_clean_scrub_stamp":"2023-10-30T17:55:06.173670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:41.569418+0000","scrub_duration":0.033937040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":864,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:00.277269+0000","last_change":"2023-10-30T17:32:34.658442+0000","last_active":"2023-10-30T17:55:00.277269+0000","last_peered":"2023-10-30T17:55:00.277269+0000","last_clean":"2023-10-30T17:55:00.277269+0000","last_became_active":"2023-10-30T17:32:34.631494+0000","last_became_peered":"2023-10-30T17:32:34.631494+0000","last_unstale":"2023-10-30T17:55:00.277269+0000","last_undegraded":"2023-10-30T17:55:00.277269+0000","last_fullsized":"2023-10-30T17:55:00.277269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:36:36.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":863,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.041139+0000","last_change":"2023-10-30T17:33:46.593988+0000","last_active":"2023-10-30T17:55:01.041139+0000","last_peered":"2023-10-30T17:55:01.041139+0000","last_clean":"2023-10-30T17:55:01.041139+0000","last_became_active":"2023-10-30T17:33:46.588555+0000","last_became_peered":"2023-10-30T17:33:46.588555+0000","last_unstale":"2023-10-30T17:55:01.041139+0000","last_undegraded":"2023-10-30T17:55:01.041139+0000","last_fullsized":"2023-10-30T17:55:01.041139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:57.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":857,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.441654+0000","last_change":"2023-10-30T17:32:35.388556+0000","last_active":"2023-10-30T17:55:04.441654+0000","last_peered":"2023-10-30T17:55:04.441654+0000","last_clean":"2023-10-30T17:55:04.441654+0000","last_became_active":"2023-10-30T17:32:35.383414+0000","last_became_peered":"2023-10-30T17:32:35.383414+0000","last_unstale":"2023-10-30T17:55:04.441654+0000","last_undegraded":"2023-10-30T17:55:04.441654+0000","last_fullsized":"2023-10-30T17:55:04.441654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":876,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.037208+0000","last_change":"2023-10-30T17:33:38.185755+0000","last_active":"2023-10-30T17:55:01.037208+0000","last_peered":"2023-10-30T17:55:01.037208+0000","last_clean":"2023-10-30T17:55:01.037208+0000","last_became_active":"2023-10-30T17:33:38.176989+0000","last_became_peered":"2023-10-30T17:33:38.176989+0000","last_unstale":"2023-10-30T17:55:01.037208+0000","last_undegraded":"2023-10-30T17:55:01.037208+0000","last_fullsized":"2023-10-30T17:55:01.037208+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:25:37.689974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":883,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.024496+0000","last_change":"2023-10-30T17:32:34.318569+0000","last_active":"2023-10-30T17:55:01.024496+0000","last_peered":"2023-10-30T17:55:01.024496+0000","last_clean":"2023-10-30T17:55:01.024496+0000","last_became_active":"2023-10-30T17:32:34.287031+0000","last_became_peered":"2023-10-30T17:32:34.287031+0000","last_unstale":"2023-10-30T17:55:01.024496+0000","last_undegraded":"2023-10-30T17:55:01.024496+0000","last_fullsized":"2023-10-30T17:55:01.024496+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:55:59.080654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":875,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.042444+0000","last_change":"2023-10-30T17:33:41.123654+0000","last_active":"2023-10-30T17:55:01.042444+0000","last_peered":"2023-10-30T17:55:01.042444+0000","last_clean":"2023-10-30T17:55:01.042444+0000","last_became_active":"2023-10-30T17:33:41.116885+0000","last_became_peered":"2023-10-30T17:33:41.116885+0000","last_unstale":"2023-10-30T17:55:01.042444+0000","last_undegraded":"2023-10-30T17:55:01.042444+0000","last_fullsized":"2023-10-30T17:55:01.042444+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:51:37.700398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:09.305307+0000","last_change":"2023-10-30T17:55:09.309575+0000","last_active":"2023-10-30T17:55:09.305307+0000","last_peered":"2023-10-30T17:55:09.305307+0000","last_clean":"2023-10-30T17:55:09.305307+0000","last_became_active":"2023-10-30T17:32:35.277163+0000","last_became_peered":"2023-10-30T17:32:35.277163+0000","last_unstale":"2023-10-30T17:55:09.305307+0000","last_undegraded":"2023-10-30T17:55:09.305307+0000","last_fullsized":"2023-10-30T17:55:09.305307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:09.302612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:09.302612+0000","last_clean_scrub_stamp":"2023-10-30T17:55:09.302612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:08:21.503163+0000","scrub_duration":0.044067580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":851,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.457471+0000","last_change":"2023-10-30T17:33:43.745646+0000","last_active":"2023-10-30T17:55:04.457471+0000","last_peered":"2023-10-30T17:55:04.457471+0000","last_clean":"2023-10-30T17:55:04.457471+0000","last_became_active":"2023-10-30T17:33:43.736021+0000","last_became_peered":"2023-10-30T17:33:43.736021+0000","last_unstale":"2023-10-30T17:55:04.457471+0000","last_undegraded":"2023-10-30T17:55:04.457471+0000","last_fullsized":"2023-10-30T17:55:04.457471+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":857,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.443340+0000","last_change":"2023-10-30T17:32:35.628758+0000","last_active":"2023-10-30T17:55:04.443340+0000","last_peered":"2023-10-30T17:55:04.443340+0000","last_clean":"2023-10-30T17:55:04.443340+0000","last_became_active":"2023-10-30T17:32:35.623395+0000","last_became_peered":"2023-10-30T17:32:35.623395+0000","last_unstale":"2023-10-30T17:55:04.443340+0000","last_undegraded":"2023-10-30T17:55:04.443340+0000","last_fullsized":"2023-10-30T17:55:04.443340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:32:32.679287+0000","last_clean_scrub_stamp":"2023-10-30T17:32:32.679287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"914'78","reported_seq":978,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:10.218608+0000","last_change":"2023-10-30T17:55:10.220901+0000","last_active":"2023-10-30T17:55:10.218608+0000","last_peered":"2023-10-30T17:55:10.218608+0000","last_clean":"2023-10-30T17:55:10.218608+0000","last_became_active":"2023-10-30T17:32:28.686958+0000","last_became_peered":"2023-10-30T17:32:28.686958+0000","last_unstale":"2023-10-30T17:55:10.218608+0000","last_undegraded":"2023-10-30T17:55:10.218608+0000","last_fullsized":"2023-10-30T17:55:10.218608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'78","last_scrub_stamp":"2023-10-30T17:55:10.216301+0000","last_deep_scrub":"914'78","last_deep_scrub_stamp":"2023-10-30T17:55:10.216301+0000","last_clean_scrub_stamp":"2023-10-30T17:55:10.216301+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:16:00.152248+0000","scrub_duration":0.12796829700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":869,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:07.218625+0000","last_change":"2023-10-30T17:55:07.219619+0000","last_active":"2023-10-30T17:55:07.218625+0000","last_peered":"2023-10-30T17:55:07.218625+0000","last_clean":"2023-10-30T17:55:07.218625+0000","last_became_active":"2023-10-30T17:33:40.602436+0000","last_became_peered":"2023-10-30T17:33:40.602436+0000","last_unstale":"2023-10-30T17:55:07.218625+0000","last_undegraded":"2023-10-30T17:55:07.218625+0000","last_fullsized":"2023-10-30T17:55:07.218625+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:07.217190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:07.217190+0000","last_clean_scrub_stamp":"2023-10-30T17:55:07.217190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:25.972202+0000","scrub_duration":0.033129934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":852,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.451334+0000","last_change":"2023-10-30T17:33:38.623594+0000","last_active":"2023-10-30T17:55:04.451334+0000","last_peered":"2023-10-30T17:55:04.451334+0000","last_clean":"2023-10-30T17:55:04.451334+0000","last_became_active":"2023-10-30T17:33:38.616054+0000","last_became_peered":"2023-10-30T17:33:38.616054+0000","last_unstale":"2023-10-30T17:55:04.451334+0000","last_undegraded":"2023-10-30T17:55:04.451334+0000","last_fullsized":"2023-10-30T17:55:04.451334+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":856,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.947540+0000","last_change":"2023-10-30T17:33:40.742636+0000","last_active":"2023-10-30T17:55:04.947540+0000","last_peered":"2023-10-30T17:55:04.947540+0000","last_clean":"2023-10-30T17:55:04.947540+0000","last_became_active":"2023-10-30T17:33:40.735511+0000","last_became_peered":"2023-10-30T17:33:40.735511+0000","last_unstale":"2023-10-30T17:55:04.947540+0000","last_undegraded":"2023-10-30T17:55:04.947540+0000","last_fullsized":"2023-10-30T17:55:04.947540+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":858,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:00.275678+0000","last_change":"2023-10-30T17:33:39.276657+0000","last_active":"2023-10-30T17:55:00.275678+0000","last_peered":"2023-10-30T17:55:00.275678+0000","last_clean":"2023-10-30T17:55:00.275678+0000","last_became_active":"2023-10-30T17:33:39.228561+0000","last_became_peered":"2023-10-30T17:33:39.228561+0000","last_unstale":"2023-10-30T17:55:00.275678+0000","last_undegraded":"2023-10-30T17:55:00.275678+0000","last_fullsized":"2023-10-30T17:55:00.275678+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:01.276873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":875,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.039874+0000","last_change":"2023-10-30T17:33:41.611205+0000","last_active":"2023-10-30T17:55:01.039874+0000","last_peered":"2023-10-30T17:55:01.039874+0000","last_clean":"2023-10-30T17:55:01.039874+0000","last_became_active":"2023-10-30T17:33:41.605208+0000","last_became_peered":"2023-10-30T17:33:41.605208+0000","last_unstale":"2023-10-30T17:55:01.039874+0000","last_undegraded":"2023-10-30T17:55:01.039874+0000","last_fullsized":"2023-10-30T17:55:01.039874+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:09.587584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.282299+0000","last_change":"2023-10-30T17:33:44.441890+0000","last_active":"2023-10-30T17:55:04.282299+0000","last_peered":"2023-10-30T17:55:04.282299+0000","last_clean":"2023-10-30T17:55:04.282299+0000","last_became_active":"2023-10-30T17:33:44.434104+0000","last_became_peered":"2023-10-30T17:33:44.434104+0000","last_unstale":"2023-10-30T17:55:04.282299+0000","last_undegraded":"2023-10-30T17:55:04.282299+0000","last_fullsized":"2023-10-30T17:55:04.282299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":859,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:00.273513+0000","last_change":"2023-10-30T17:33:43.330759+0000","last_active":"2023-10-30T17:55:00.273513+0000","last_peered":"2023-10-30T17:55:00.273513+0000","last_clean":"2023-10-30T17:55:00.273513+0000","last_became_active":"2023-10-30T17:33:43.322860+0000","last_became_peered":"2023-10-30T17:33:43.322860+0000","last_unstale":"2023-10-30T17:55:00.273513+0000","last_undegraded":"2023-10-30T17:55:00.273513+0000","last_fullsized":"2023-10-30T17:55:00.273513+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:56:01.616064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.283983+0000","last_change":"2023-10-30T17:33:53.140054+0000","last_active":"2023-10-30T17:55:04.283983+0000","last_peered":"2023-10-30T17:55:04.283983+0000","last_clean":"2023-10-30T17:55:04.283983+0000","last_became_active":"2023-10-30T17:33:53.130615+0000","last_became_peered":"2023-10-30T17:33:53.130615+0000","last_unstale":"2023-10-30T17:55:04.283983+0000","last_undegraded":"2023-10-30T17:55:04.283983+0000","last_fullsized":"2023-10-30T17:55:04.283983+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":856,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.931367+0000","last_change":"2023-10-30T17:33:37.571189+0000","last_active":"2023-10-30T17:55:04.931367+0000","last_peered":"2023-10-30T17:55:04.931367+0000","last_clean":"2023-10-30T17:55:04.931367+0000","last_became_active":"2023-10-30T17:33:37.540749+0000","last_became_peered":"2023-10-30T17:33:37.540749+0000","last_unstale":"2023-10-30T17:55:04.931367+0000","last_undegraded":"2023-10-30T17:55:04.931367+0000","last_fullsized":"2023-10-30T17:55:04.931367+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":851,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.454033+0000","last_change":"2023-10-30T17:33:58.392321+0000","last_active":"2023-10-30T17:55:04.454033+0000","last_peered":"2023-10-30T17:55:04.454033+0000","last_clean":"2023-10-30T17:55:04.454033+0000","last_became_active":"2023-10-30T17:33:58.383320+0000","last_became_peered":"2023-10-30T17:33:58.383320+0000","last_unstale":"2023-10-30T17:55:04.454033+0000","last_undegraded":"2023-10-30T17:55:04.454033+0000","last_fullsized":"2023-10-30T17:55:04.454033+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":860,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.052332+0000","last_change":"2023-10-30T17:33:38.595237+0000","last_active":"2023-10-30T17:55:01.052332+0000","last_peered":"2023-10-30T17:55:01.052332+0000","last_clean":"2023-10-30T17:55:01.052332+0000","last_became_active":"2023-10-30T17:33:38.589479+0000","last_became_peered":"2023-10-30T17:33:38.589479+0000","last_unstale":"2023-10-30T17:55:01.052332+0000","last_undegraded":"2023-10-30T17:55:01.052332+0000","last_fullsized":"2023-10-30T17:55:01.052332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T03:17:44.312113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":862,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-10-30T17:55:01.042829+0000","last_change":"2023-10-30T17:33:40.505225+0000","last_active":"2023-10-30T17:55:01.042829+0000","last_peered":"2023-10-30T17:55:01.042829+0000","last_clean":"2023-10-30T17:55:01.042829+0000","last_became_active":"2023-10-30T17:33:40.476315+0000","last_became_peered":"2023-10-30T17:33:40.476315+0000","last_unstale":"2023-10-30T17:55:01.042829+0000","last_undegraded":"2023-10-30T17:55:01.042829+0000","last_fullsized":"2023-10-30T17:55:01.042829+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:57:02.608925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":851,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.447939+0000","last_change":"2023-10-30T17:33:38.803571+0000","last_active":"2023-10-30T17:55:04.447939+0000","last_peered":"2023-10-30T17:55:04.447939+0000","last_clean":"2023-10-30T17:55:04.447939+0000","last_became_active":"2023-10-30T17:33:38.795801+0000","last_became_peered":"2023-10-30T17:33:38.795801+0000","last_unstale":"2023-10-30T17:55:04.447939+0000","last_undegraded":"2023-10-30T17:55:04.447939+0000","last_fullsized":"2023-10-30T17:55:04.447939+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":17,"seq":73014444304,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60908,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310932,"statfs":{"total":96636764160,"available":96574394368,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":7.8239999999999998,"5min":14.673999999999999,"15min":20.202999999999999},"min":{"1min":4.2460000000000004,"5min":4.2460000000000004,"15min":4.2460000000000004},"max":{"1min":15.343,"5min":217.97399999999999,"15min":296.49400000000003},"last":12.343999999999999},{"interface":"front","average":{"1min":13.112,"5min":16.077000000000002,"15min":21.355},"min":{"1min":7.6120000000000001,"5min":5.6189999999999998,"15min":4.6299999999999999},"max":{"1min":31.678999999999998,"5min":134.833,"15min":276.49799999999999},"last":7.7990000000000004}]},{"osd":2,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":16.689,"5min":19.114999999999998,"15min":26.032},"min":{"1min":6.319,"5min":5.915,"15min":5.915},"max":{"1min":101.64100000000001,"5min":218.22399999999999,"15min":591.02499999999998},"last":22.922999999999998},{"interface":"front","average":{"1min":19.178999999999998,"5min":20.963000000000001,"15min":25.277000000000001},"min":{"1min":9.0850000000000009,"5min":7.6520000000000001,"15min":6.4930000000000003},"max":{"1min":103.506,"5min":228.36500000000001,"15min":259.52300000000002},"last":10.396000000000001}]},{"osd":3,"last update":"Mon Oct 30 17:54:58 2023","interfaces":[{"interface":"back","average":{"1min":13.741,"5min":75.924000000000007,"15min":40.366999999999997},"min":{"1min":6.4290000000000003,"5min":6.4290000000000003,"15min":5.8099999999999996},"max":{"1min":62.359000000000002,"5min":5036.1790000000001,"15min":5036.1790000000001},"last":6.8730000000000002},{"interface":"front","average":{"1min":17.495999999999999,"5min":79.316999999999993,"15min":44.966999999999999},"min":{"1min":7.6950000000000003,"5min":7.2770000000000001,"15min":6.4909999999999997},"max":{"1min":47.174999999999997,"5min":5057.1229999999996,"15min":5057.1229999999996},"last":23.972000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":24.963000000000001,"5min":22.420999999999999,"15min":22.024000000000001},"min":{"1min":7.3259999999999996,"5min":7.3259999999999996,"15min":6.3689999999999998},"max":{"1min":159.94399999999999,"5min":204.91,"15min":281.16399999999999},"last":11.340999999999999},{"interface":"front","average":{"1min":29.393999999999998,"5min":26.954000000000001,"15min":27.579000000000001},"min":{"1min":9.0530000000000008,"5min":9.0530000000000008,"15min":5.8879999999999999},"max":{"1min":190.24199999999999,"5min":218.85300000000001,"15min":297.57299999999998},"last":25.719000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":19.146999999999998,"5min":25.224,"15min":28.202999999999999},"min":{"1min":8.6660000000000004,"5min":8.4960000000000004,"15min":7.9370000000000003},"max":{"1min":44.509,"5min":260.81099999999998,"15min":260.81099999999998},"last":9.1069999999999993},{"interface":"front","average":{"1min":15.294,"5min":22.741,"15min":25.870000000000001},"min":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.1189999999999998},"max":{"1min":32.457999999999998,"5min":236.83500000000001,"15min":236.83500000000001},"last":8.5180000000000007}]},{"osd":6,"last update":"Mon Oct 30 17:54:23 2023","interfaces":[{"interface":"back","average":{"1min":19.766999999999999,"5min":22.66,"15min":27.329000000000001},"min":{"1min":10.051,"5min":9.968,"15min":8.4019999999999992},"max":{"1min":37.848999999999997,"5min":125.158,"15min":292.47500000000002},"last":14.300000000000001},{"interface":"front","average":{"1min":20.893000000000001,"5min":26.652000000000001,"15min":28.780000000000001},"min":{"1min":11.971,"5min":6.8449999999999998,"15min":6.8449999999999998},"max":{"1min":62.518000000000001,"5min":189.01599999999999,"15min":273.64999999999998},"last":13.108000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":22.042999999999999,"5min":25.934999999999999,"15min":29.561},"min":{"1min":12.084,"5min":10.622999999999999,"15min":9.157},"max":{"1min":60.759,"5min":204.12200000000001,"15min":262.74200000000002},"last":15.457000000000001},{"interface":"front","average":{"1min":26.411999999999999,"5min":31.727,"15min":33.009999999999998},"min":{"1min":13.321,"5min":12.188000000000001,"15min":10.122999999999999},"max":{"1min":100.953,"5min":219.833,"15min":264.81},"last":29.015999999999998}]}]},{"osd":3,"up_from":16,"seq":68719477000,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83120,"kb_used_data":7528,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288720,"statfs":{"total":96636764160,"available":96551649280,"internally_reserved":0,"allocated":7708672,"data_stored":4236976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":30.98,"5min":20.527999999999999,"15min":16.279},"min":{"1min":5.5099999999999998,"5min":4.4489999999999998,"15min":4.3869999999999996},"max":{"1min":184.12899999999999,"5min":203.85499999999999,"15min":203.85499999999999},"last":11.852},{"interface":"front","average":{"1min":33.973999999999997,"5min":21.707999999999998,"15min":18.739999999999998},"min":{"1min":6.4829999999999997,"5min":4.8449999999999998,"15min":4.367},"max":{"1min":138.00399999999999,"5min":176.83500000000001,"15min":364.05399999999997},"last":9.5950000000000006}]},{"osd":1,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":20.564,"5min":23.222000000000001,"15min":23.289000000000001},"min":{"1min":7.4489999999999998,"5min":7.1909999999999998,"15min":5.5380000000000003},"max":{"1min":124.999,"5min":178.04300000000001,"15min":190.55199999999999},"last":16.806999999999999},{"interface":"front","average":{"1min":19.936,"5min":23.527000000000001,"15min":22.817},"min":{"1min":7.7690000000000001,"5min":5.9290000000000003,"15min":5.609},"max":{"1min":105.627,"5min":210.381,"15min":240.64500000000001},"last":14.972}]},{"osd":2,"last update":"Mon Oct 30 17:54:19 2023","interfaces":[{"interface":"back","average":{"1min":34.308999999999997,"5min":27.562000000000001,"15min":24.027999999999999},"min":{"1min":7.2969999999999997,"5min":7.2489999999999997,"15min":5.234},"max":{"1min":251.941,"5min":251.941,"15min":397.50099999999998},"last":8.7880000000000003},{"interface":"front","average":{"1min":19.899000000000001,"5min":22.161999999999999,"15min":25.760999999999999},"min":{"1min":7.1539999999999999,"5min":6.54,"15min":6.54},"max":{"1min":85.495000000000005,"5min":147.50399999999999,"15min":219.65000000000001},"last":7.7869999999999999}]},{"osd":4,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":24.489999999999998,"5min":34.338000000000001,"15min":26.545000000000002},"min":{"1min":9.0999999999999996,"5min":7.7729999999999997,"15min":5.8140000000000001},"max":{"1min":126.723,"5min":318.92200000000003,"15min":318.92200000000003},"last":12.568},{"interface":"front","average":{"1min":26.800000000000001,"5min":27.832999999999998,"15min":21.242999999999999},"min":{"1min":8.5909999999999993,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":137.06999999999999,"5min":268.01600000000002,"15min":268.01600000000002},"last":10.895}]},{"osd":5,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":43.463000000000001,"5min":26.356000000000002,"15min":23.449000000000002},"min":{"1min":10.387,"5min":7.7569999999999997,"15min":7.7569999999999997},"max":{"1min":217.596,"5min":217.596,"15min":221.821},"last":18.971},{"interface":"front","average":{"1min":35.868000000000002,"5min":30.478999999999999,"15min":26.512},"min":{"1min":9.6460000000000008,"5min":7.2009999999999996,"15min":7.2009999999999996},"max":{"1min":223.203,"5min":225.15299999999999,"15min":225.15299999999999},"last":13.868}]},{"osd":6,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":33.683999999999997,"5min":31.859000000000002,"15min":26.734000000000002},"min":{"1min":11.108000000000001,"5min":7.9589999999999996,"15min":7.7439999999999998},"max":{"1min":137.48599999999999,"5min":272.536,"15min":272.536},"last":17.916},{"interface":"front","average":{"1min":30.719000000000001,"5min":24.899999999999999,"15min":23.899000000000001},"min":{"1min":9.8620000000000001,"5min":9.3040000000000003,"15min":7.524},"max":{"1min":121.95999999999999,"5min":162.61799999999999,"15min":199.892},"last":16.254000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":28.047000000000001,"5min":30.155000000000001,"15min":28.039000000000001},"min":{"1min":12.372999999999999,"5min":9.8399999999999999,"15min":9.7059999999999995},"max":{"1min":124.315,"5min":196.05699999999999,"15min":196.66399999999999},"last":19.786000000000001},{"interface":"front","average":{"1min":25.338000000000001,"5min":29.327999999999999,"15min":28.225000000000001},"min":{"1min":8.7590000000000003,"5min":8.7590000000000003,"15min":8.7590000000000003},"max":{"1min":157.10499999999999,"5min":182.07499999999999,"15min":221.59100000000001},"last":14.426}]}]},{"osd":1,"up_from":16,"seq":68719477011,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61300,"kb_used_data":7916,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310540,"statfs":{"total":96636764160,"available":96573992960,"internally_reserved":0,"allocated":8105984,"data_stored":4634811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":10.568,"5min":13.151999999999999,"15min":17.620999999999999},"min":{"1min":5.1379999999999999,"5min":4.8099999999999996,"15min":3.8279999999999998},"max":{"1min":26.812000000000001,"5min":162.364,"15min":313.69400000000002},"last":10.185},{"interface":"front","average":{"1min":13.189,"5min":13.273999999999999,"15min":17.713000000000001},"min":{"1min":8.4429999999999996,"5min":5.4560000000000004,"15min":4.9089999999999998},"max":{"1min":21.195,"5min":59.277000000000001,"15min":209.62799999999999},"last":14.846}]},{"osd":2,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":17.059999999999999,"5min":16.346,"15min":21.375},"min":{"1min":8.5259999999999998,"5min":6.2190000000000003,"15min":6.0279999999999996},"max":{"1min":84.930999999999997,"5min":95.995000000000005,"15min":267.83499999999998},"last":16.465},{"interface":"front","average":{"1min":19.794,"5min":18.084,"15min":20.032},"min":{"1min":11.779999999999999,"5min":7.0270000000000001,"15min":5.8550000000000004},"max":{"1min":89.875,"5min":89.875,"15min":178.44},"last":19.327000000000002}]},{"osd":3,"last update":"Mon Oct 30 17:54:17 2023","interfaces":[{"interface":"back","average":{"1min":10.504,"5min":246.75800000000001,"15min":96.697000000000003},"min":{"1min":6.1180000000000003,"5min":6.1050000000000004,"15min":5.2990000000000004},"max":{"1min":31.059000000000001,"5min":8788.2559999999994,"15min":8788.2559999999994},"last":11.08},{"interface":"front","average":{"1min":17.736999999999998,"5min":247.22,"15min":98.808000000000007},"min":{"1min":11.208,"5min":6.8819999999999997,"15min":6.8819999999999997},"max":{"1min":29.475999999999999,"5min":8677.7369999999992,"15min":8677.7369999999992},"last":17.388000000000002}]},{"osd":4,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":15.772,"5min":17.181999999999999,"15min":23.558},"min":{"1min":7.7359999999999998,"5min":6.8220000000000001,"15min":6.8220000000000001},"max":{"1min":91.501999999999995,"5min":91.501999999999995,"15min":235.125},"last":9.516},{"interface":"front","average":{"1min":20.797999999999998,"5min":21.347000000000001,"15min":23.934000000000001},"min":{"1min":10.278,"5min":7.673,"15min":7.1829999999999998},"max":{"1min":92.772000000000006,"5min":92.772000000000006,"15min":334.19900000000001},"last":12.852}]},{"osd":5,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":15.865,"5min":19.510000000000002,"15min":27.055},"min":{"1min":9.0389999999999997,"5min":9.0389999999999997,"15min":7.6360000000000001},"max":{"1min":43.468000000000004,"5min":89.837000000000003,"15min":220.65799999999999},"last":15.651999999999999},{"interface":"front","average":{"1min":21.504999999999999,"5min":21.619,"15min":31.059999999999999},"min":{"1min":14.56,"5min":8.0869999999999997,"15min":8.0869999999999997},"max":{"1min":50.332000000000001,"5min":134.42599999999999,"15min":262.61799999999999},"last":18.327999999999999}]},{"osd":6,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":10.253,"5min":15.454000000000001,"15min":24.402000000000001},"min":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":6.8769999999999998},"max":{"1min":20.927,"5min":200.369,"15min":249.25299999999999},"last":13.996},{"interface":"front","average":{"1min":17.808,"5min":20.117999999999999,"15min":21.41},"min":{"1min":10.965,"5min":8.8610000000000007,"15min":6.2880000000000003},"max":{"1min":28.323,"5min":201.756,"15min":259.03899999999999},"last":23.574999999999999}]},{"osd":7,"last update":"Mon Oct 30 17:54:07 2023","interfaces":[{"interface":"back","average":{"1min":17.879000000000001,"5min":21.974,"15min":27.113},"min":{"1min":11.935,"5min":9.8200000000000003,"15min":9.4269999999999996},"max":{"1min":29.427,"5min":100.73999999999999,"15min":200.62100000000001},"last":20.268000000000001},{"interface":"front","average":{"1min":21.350999999999999,"5min":26.129999999999999,"15min":30.661000000000001},"min":{"1min":16.481000000000002,"5min":10.682,"15min":10.115},"max":{"1min":36.606999999999999,"5min":203.929,"15min":205.61799999999999},"last":22.596}]}]},{"osd":2,"up_from":15,"seq":64424509714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60912,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310928,"statfs":{"total":96636764160,"available":96574390272,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":14.316000000000001,"5min":13.782,"15min":18.59},"min":{"1min":5.3460000000000001,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":106.839,"5min":106.839,"15min":334.36900000000003},"last":13.116},{"interface":"front","average":{"1min":18.443000000000001,"5min":17.509,"15min":21.878},"min":{"1min":7.7990000000000004,"5min":7.1319999999999997,"15min":6.7489999999999997},"max":{"1min":111.07599999999999,"5min":111.07599999999999,"15min":283.46899999999999},"last":17.562999999999999}]},{"osd":1,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":25.234000000000002,"5min":23.196999999999999,"15min":24.32},"min":{"1min":9.1259999999999994,"5min":6.1539999999999999,"15min":4.7830000000000004},"max":{"1min":122.607,"5min":194.499,"15min":273.93599999999998},"last":14.944000000000001},{"interface":"front","average":{"1min":18.93,"5min":14.327999999999999,"15min":15.942},"min":{"1min":5.0069999999999997,"5min":4.9020000000000001,"15min":4.1890000000000001},"max":{"1min":109.767,"5min":109.767,"15min":152.87299999999999},"last":8.1669999999999998}]},{"osd":3,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":12.736000000000001,"5min":219.62799999999999,"15min":85.793999999999997},"min":{"1min":5.6429999999999998,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":62.838000000000001,"5min":8749.8639999999996,"15min":8749.8639999999996},"last":7.8419999999999996},{"interface":"front","average":{"1min":19.050999999999998,"5min":225.965,"15min":91.936000000000007},"min":{"1min":9.2469999999999999,"5min":7.7859999999999996,"15min":6.6799999999999997},"max":{"1min":75.923000000000002,"5min":8752.7450000000008,"15min":8752.7450000000008},"last":12.085000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":24.823,"5min":22.719000000000001,"15min":28.533999999999999},"min":{"1min":8.8420000000000005,"5min":8.5990000000000002,"15min":8.5990000000000002},"max":{"1min":115.70999999999999,"5min":124.502,"15min":301.34399999999999},"last":16.704999999999998},{"interface":"front","average":{"1min":27.699999999999999,"5min":27.585000000000001,"15min":32.787999999999997},"min":{"1min":11.974,"5min":11.974,"15min":11.824},"max":{"1min":117.285,"5min":134.44800000000001,"15min":300.68900000000002},"last":18.367000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:54:15 2023","interfaces":[{"interface":"back","average":{"1min":21.994,"5min":19.698,"15min":29.402999999999999},"min":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":124.914,"5min":198.46700000000001,"15min":318.01900000000001},"last":13.765000000000001},{"interface":"front","average":{"1min":21.603999999999999,"5min":18.942,"15min":24.146000000000001},"min":{"1min":10.16,"5min":7.5960000000000001,"15min":5.8920000000000003},"max":{"1min":122.861,"5min":122.861,"15min":325.87099999999998},"last":15.582000000000001}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":19.571000000000002,"5min":18.742000000000001,"15min":20.893000000000001},"min":{"1min":8.9290000000000003,"5min":8.7420000000000009,"15min":6.3979999999999997},"max":{"1min":70.927999999999997,"5min":70.927999999999997,"15min":328.983},"last":9.8330000000000002},{"interface":"front","average":{"1min":25.178000000000001,"5min":26.393999999999998,"15min":30.449000000000002},"min":{"1min":11.391,"5min":11.391,"15min":10.986000000000001},"max":{"1min":124.36799999999999,"5min":133.02199999999999,"15min":306.81799999999998},"last":19.312000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":17.791,"5min":16.997,"15min":22.556000000000001},"min":{"1min":7.8630000000000004,"5min":7.2549999999999999,"15min":6.9669999999999996},"max":{"1min":127.804,"5min":127.804,"15min":307.69499999999999},"last":10.401999999999999},{"interface":"front","average":{"1min":21.841999999999999,"5min":22.962,"15min":28.602},"min":{"1min":10.927,"5min":8.9550000000000001,"15min":8.5700000000000003},"max":{"1min":129.26599999999999,"5min":129.26599999999999,"15min":317.13400000000001},"last":11.336}]}]},{"osd":7,"up_from":14,"seq":60129542417,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83124,"kb_used_data":7528,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288716,"statfs":{"total":96636764160,"available":96551645184,"internally_reserved":0,"allocated":7708672,"data_stored":4236976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77396427},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":18.553000000000001,"5min":17.256,"15min":18.238},"min":{"1min":4.6230000000000002,"5min":4.6230000000000002,"15min":4.1580000000000004},"max":{"1min":172.93199999999999,"5min":179.608,"15min":259.12200000000001},"last":15.784000000000001},{"interface":"front","average":{"1min":19.984999999999999,"5min":18.443999999999999,"15min":18.128},"min":{"1min":4.5999999999999996,"5min":4.5999999999999996,"15min":4.3959999999999999},"max":{"1min":176.59399999999999,"5min":176.59399999999999,"15min":258.25799999999998},"last":6.7599999999999998}]},{"osd":1,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":13.958,"5min":15.481,"15min":18.420999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.2939999999999996},"max":{"1min":50.183,"5min":83.355000000000004,"15min":249.69900000000001},"last":10.082000000000001},{"interface":"front","average":{"1min":16.277000000000001,"5min":19.504000000000001,"15min":24.225999999999999},"min":{"1min":8.0890000000000004,"5min":8.0640000000000001,"15min":5.0439999999999996},"max":{"1min":67.171000000000006,"5min":199.15299999999999,"15min":334.67599999999999},"last":9.5589999999999993}]},{"osd":2,"last update":"Mon Oct 30 17:55:05 2023","interfaces":[{"interface":"back","average":{"1min":20.262,"5min":18.870999999999999,"15min":20.088000000000001},"min":{"1min":7.7910000000000004,"5min":6.3410000000000002,"15min":6.3140000000000001},"max":{"1min":145.50800000000001,"5min":145.50800000000001,"15min":403.90899999999999},"last":10.680999999999999},{"interface":"front","average":{"1min":26.34,"5min":26.309999999999999,"15min":25.690999999999999},"min":{"1min":9.641,"5min":8.4559999999999995,"15min":7.6829999999999998},"max":{"1min":146.65100000000001,"5min":276.67700000000002,"15min":354.34800000000001},"last":16.376000000000001}]},{"osd":3,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":19.672000000000001,"5min":165.75800000000001,"15min":72.480000000000004},"min":{"1min":8.8859999999999992,"5min":8.8859999999999992,"15min":6.226},"max":{"1min":71.772999999999996,"5min":8740.9120000000003,"15min":8740.9120000000003},"last":11.717000000000001},{"interface":"front","average":{"1min":16.125,"5min":162.107,"15min":70.201999999999998},"min":{"1min":7.444,"5min":7.3840000000000003,"15min":6.1120000000000001},"max":{"1min":50.947000000000003,"5min":8827.2800000000007,"15min":8827.2800000000007},"last":13.077}]},{"osd":4,"last update":"Mon Oct 30 17:54:11 2023","interfaces":[{"interface":"back","average":{"1min":17.946999999999999,"5min":23.007999999999999,"15min":24.891999999999999},"min":{"1min":6.758,"5min":6.758,"15min":6.758},"max":{"1min":73.082999999999998,"5min":193.517,"15min":265.78300000000002},"last":7.4109999999999996},{"interface":"front","average":{"1min":25.338000000000001,"5min":28.114999999999998,"15min":26.849},"min":{"1min":11.759,"5min":10.228999999999999,"15min":8.5839999999999996},"max":{"1min":121.331,"5min":204.27799999999999,"15min":297.99900000000002},"last":14.880000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:55:05 2023","interfaces":[{"interface":"back","average":{"1min":28.478000000000002,"5min":24.763999999999999,"15min":27.021999999999998},"min":{"1min":11.518000000000001,"5min":9.4339999999999993,"15min":7.7729999999999997},"max":{"1min":173.75999999999999,"5min":235.08500000000001,"15min":298.197},"last":17.573},{"interface":"front","average":{"1min":19.510000000000002,"5min":21.577000000000002,"15min":21.898},"min":{"1min":10.781000000000001,"5min":6.5010000000000003,"15min":5.5010000000000003},"max":{"1min":79.013000000000005,"5min":266.30200000000002,"15min":280.34100000000001},"last":11.012}]},{"osd":6,"last update":"Mon Oct 30 17:55:05 2023","interfaces":[{"interface":"back","average":{"1min":26.076000000000001,"5min":23.91,"15min":23.134},"min":{"1min":8.6750000000000007,"5min":8.6750000000000007,"15min":6.9400000000000004},"max":{"1min":122.34099999999999,"5min":174.517,"15min":260.10399999999998},"last":13.821999999999999},{"interface":"front","average":{"1min":25.681999999999999,"5min":26.704999999999998,"15min":26.986999999999998},"min":{"1min":8.7430000000000003,"5min":8.7430000000000003,"15min":8.0060000000000002},"max":{"1min":79.855000000000004,"5min":140.06399999999999,"15min":257.62599999999998},"last":18.489999999999998}]}]},{"osd":6,"up_from":14,"seq":60129542420,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78960,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292880,"statfs":{"total":96636764160,"available":96555909120,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":9.3670000000000009,"5min":16.529,"15min":18.324000000000002},"min":{"1min":4.7869999999999999,"5min":4.3339999999999996,"15min":4.1840000000000002},"max":{"1min":26.079000000000001,"5min":135.98099999999999,"15min":254.899},"last":5.0830000000000002},{"interface":"front","average":{"1min":11.316000000000001,"5min":17.039999999999999,"15min":19.542000000000002},"min":{"1min":6.4569999999999999,"5min":5.4219999999999997,"15min":5.4219999999999997},"max":{"1min":28.175999999999998,"5min":97.100999999999999,"15min":270.51600000000002},"last":10.198}]},{"osd":1,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":19.887,"5min":20.942,"15min":20.899999999999999},"min":{"1min":7.5099999999999998,"5min":7.2350000000000003,"15min":5.1120000000000001},"max":{"1min":138.97300000000001,"5min":149.53899999999999,"15min":177.15799999999999},"last":16.969000000000001},{"interface":"front","average":{"1min":12.612,"5min":20.626000000000001,"15min":23.800999999999998},"min":{"1min":7.5529999999999999,"5min":6.2199999999999998,"15min":5.883},"max":{"1min":28.498000000000001,"5min":221.483,"15min":309.04199999999997},"last":10.955}]},{"osd":2,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":14.561999999999999,"5min":19.271000000000001,"15min":20.66},"min":{"1min":7.1379999999999999,"5min":6.8929999999999998,"15min":5.859},"max":{"1min":31.204000000000001,"5min":122.018,"15min":231.15600000000001},"last":11.941000000000001},{"interface":"front","average":{"1min":16.201000000000001,"5min":24.071999999999999,"15min":23.096},"min":{"1min":11.792,"5min":11.173999999999999,"15min":6.9889999999999999},"max":{"1min":26.43,"5min":149.17500000000001,"15min":197.58699999999999},"last":13.486000000000001}]},{"osd":3,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":9.8900000000000006,"5min":253.59399999999999,"15min":99.061000000000007},"min":{"1min":6.6689999999999996,"5min":5.5039999999999996,"15min":5.3849999999999998},"max":{"1min":14.733000000000001,"5min":7958.2420000000002,"15min":7958.2420000000002},"last":5.9909999999999997},{"interface":"front","average":{"1min":13.827,"5min":254.506,"15min":99.292000000000002},"min":{"1min":8.7859999999999996,"5min":6.7009999999999996,"15min":5.9859999999999998},"max":{"1min":23.07,"5min":7971.3900000000003,"15min":7971.3900000000003},"last":12.819000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":15.43,"5min":25.451000000000001,"15min":25.949000000000002},"min":{"1min":10.375999999999999,"5min":7.5300000000000002,"15min":7.5300000000000002},"max":{"1min":22.405000000000001,"5min":236.994,"15min":262.15699999999998},"last":14.718999999999999},{"interface":"front","average":{"1min":16.157,"5min":23.646999999999998,"15min":23.992000000000001},"min":{"1min":9.6839999999999993,"5min":8.6479999999999997,"15min":6.7859999999999996},"max":{"1min":23.186,"5min":228.88200000000001,"15min":228.88200000000001},"last":14.145}]},{"osd":5,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":11.866,"5min":20.151,"15min":24.649999999999999},"min":{"1min":6.5540000000000003,"5min":6.2359999999999998,"15min":5.7809999999999997},"max":{"1min":18.652000000000001,"5min":191.154,"15min":289.22699999999998},"last":8.1969999999999992},{"interface":"front","average":{"1min":17.635000000000002,"5min":25.885999999999999,"15min":27.363},"min":{"1min":14.632999999999999,"5min":10.679,"15min":7.835},"max":{"1min":25.047000000000001,"5min":230.613,"15min":230.613},"last":15.997}]},{"osd":7,"last update":"Mon Oct 30 17:54:09 2023","interfaces":[{"interface":"back","average":{"1min":27.414000000000001,"5min":24.706,"15min":26.167999999999999},"min":{"1min":9.2230000000000008,"5min":8.0370000000000008,"15min":7.2729999999999997},"max":{"1min":164.86600000000001,"5min":192.72399999999999,"15min":290.22300000000001},"last":7.1609999999999996},{"interface":"front","average":{"1min":27.791,"5min":26.841000000000001,"15min":28.561},"min":{"1min":12.119999999999999,"5min":9.5440000000000005,"15min":7.4489999999999998},"max":{"1min":130.845,"5min":270.01900000000001,"15min":270.01900000000001},"last":17.960999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542420,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60912,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310928,"statfs":{"total":96636764160,"available":96574390272,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":8.5139999999999993,"5min":21.638000000000002,"15min":19.297000000000001},"min":{"1min":4.5659999999999998,"5min":3.7290000000000001,"15min":3.7290000000000001},"max":{"1min":31.369,"5min":278.70299999999997,"15min":317.529},"last":6.0350000000000001},{"interface":"front","average":{"1min":8.5500000000000007,"5min":19.698,"15min":19.140999999999998},"min":{"1min":4.9850000000000003,"5min":4.9850000000000003,"15min":4.8040000000000003},"max":{"1min":32.368000000000002,"5min":174.08099999999999,"15min":200.29300000000001},"last":11.874000000000001}]},{"osd":1,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":14.672000000000001,"5min":18,"15min":23.431999999999999},"min":{"1min":5.6029999999999998,"5min":5.6029999999999998,"15min":5.6029999999999998},"max":{"1min":25.289999999999999,"5min":136.81200000000001,"15min":307.86799999999999},"last":18.177},{"interface":"front","average":{"1min":12.586,"5min":18.454999999999998,"15min":23.405000000000001},"min":{"1min":7.9080000000000004,"5min":6.1669999999999998,"15min":6.0819999999999999},"max":{"1min":30.154,"5min":139.191,"15min":307.05500000000001},"last":8.0999999999999996}]},{"osd":2,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":14.393000000000001,"5min":19.882999999999999,"15min":24.52},"min":{"1min":7.2400000000000002,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":25.452999999999999,"5min":279.37099999999998,"15min":283.87599999999998},"last":9.6809999999999992},{"interface":"front","average":{"1min":24.056000000000001,"5min":21.108000000000001,"15min":22.945},"min":{"1min":9.3710000000000004,"5min":6.9390000000000001,"15min":5.8259999999999996},"max":{"1min":198.02500000000001,"5min":198.02500000000001,"15min":321.54000000000002},"last":18.82}]},{"osd":3,"last update":"Mon Oct 30 17:54:12 2023","interfaces":[{"interface":"back","average":{"1min":27.785,"5min":165.78,"15min":72.799000000000007},"min":{"1min":9.952,"5min":5.9219999999999997,"15min":5.9219999999999997},"max":{"1min":247.03399999999999,"5min":8219.1820000000007,"15min":8219.1820000000007},"last":13.919},{"interface":"front","average":{"1min":14.07,"5min":161.76900000000001,"15min":71.828000000000003},"min":{"1min":8.7919999999999998,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":34.832999999999998,"5min":8140.4719999999998,"15min":8140.4719999999998},"last":10.481999999999999}]},{"osd":5,"last update":"Mon Oct 30 17:54:53 2023","interfaces":[{"interface":"back","average":{"1min":18.893000000000001,"5min":22.899000000000001,"15min":23.814},"min":{"1min":9.1790000000000003,"5min":6.4169999999999998,"15min":6.4169999999999998},"max":{"1min":65.632000000000005,"5min":292.22800000000001,"15min":292.22800000000001},"last":13.163},{"interface":"front","average":{"1min":18.199999999999999,"5min":22.989999999999998,"15min":22.789999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":59.558,"5min":179.67500000000001,"15min":180.28800000000001},"last":12.516}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":28.52,"5min":26.338999999999999,"15min":25.452000000000002},"min":{"1min":12.355,"5min":9.7889999999999997,"15min":6.5810000000000004},"max":{"1min":191.995,"5min":191.995,"15min":293.16300000000001},"last":14.563000000000001},{"interface":"front","average":{"1min":17.751999999999999,"5min":26.558,"15min":27.359999999999999},"min":{"1min":8.516,"5min":8.2420000000000009,"15min":7.6779999999999999},"max":{"1min":43.220999999999997,"5min":207.30099999999999,"15min":281.62799999999999},"last":21.286000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":18.670000000000002,"5min":26.806999999999999,"15min":28.308},"min":{"1min":12.002000000000001,"5min":8.0289999999999999,"15min":7.4969999999999999},"max":{"1min":43.93,"5min":396.928,"15min":396.928},"last":15.512},{"interface":"front","average":{"1min":18.928000000000001,"5min":32.037999999999997,"15min":29.954999999999998},"min":{"1min":10.805999999999999,"5min":8.9809999999999999,"15min":7.0860000000000003},"max":{"1min":46.781999999999996,"5min":366.32499999999999,"15min":366.32499999999999},"last":20.202000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575124,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79352,"kb_used_data":7916,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292488,"statfs":{"total":96636764160,"available":96555507712,"internally_reserved":0,"allocated":8105984,"data_stored":4634811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.609999999999999,"5min":14.541,"15min":16.334},"min":{"1min":4.7089999999999996,"5min":3.9910000000000001,"15min":3.9569999999999999},"max":{"1min":83.933999999999997,"5min":109.919,"15min":155.58799999999999},"last":13.872},{"interface":"front","average":{"1min":17.244,"5min":16.513999999999999,"15min":18.204999999999998},"min":{"1min":7.4370000000000003,"5min":4.8789999999999996,"15min":4.6900000000000004},"max":{"1min":82.497,"5min":159.01900000000001,"15min":180.94499999999999},"last":4.6429999999999998}]},{"osd":1,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":10.683999999999999,"5min":18.123999999999999,"15min":20.596},"min":{"1min":5.0970000000000004,"5min":5.0970000000000004,"15min":4.8810000000000002},"max":{"1min":30.02,"5min":228.15199999999999,"15min":233.22800000000001},"last":11.882},{"interface":"front","average":{"1min":14.571,"5min":15.516999999999999,"15min":18.213000000000001},"min":{"1min":8.5969999999999995,"5min":5.8259999999999996,"15min":5.1120000000000001},"max":{"1min":28.818999999999999,"5min":128.197,"15min":278.41300000000001},"last":9.8710000000000004}]},{"osd":2,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":16.053999999999998,"5min":16.785,"15min":19.841000000000001},"min":{"1min":7.101,"5min":7.101,"15min":5.8949999999999996},"max":{"1min":30.600999999999999,"5min":135.75999999999999,"15min":234.09299999999999},"last":11.218},{"interface":"front","average":{"1min":13.555999999999999,"5min":15.548,"15min":20.547000000000001},"min":{"1min":6.657,"5min":6.5510000000000002,"15min":6.5060000000000002},"max":{"1min":27.719999999999999,"5min":87.316999999999993,"15min":277.52300000000002},"last":7.3819999999999997}]},{"osd":3,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":13.231999999999999,"5min":151.136,"15min":65.766999999999996},"min":{"1min":6.2080000000000002,"5min":6.2080000000000002,"15min":6.2080000000000002},"max":{"1min":48.613999999999997,"5min":6785.9989999999998,"15min":6785.9989999999998},"last":8.7210000000000001},{"interface":"front","average":{"1min":16.672999999999998,"5min":151.255,"15min":65.844999999999999},"min":{"1min":6.6870000000000003,"5min":6.6870000000000003,"15min":6.2839999999999998},"max":{"1min":54.656999999999996,"5min":6786.7420000000002,"15min":6786.7420000000002},"last":10.542999999999999}]},{"osd":4,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.928999999999998,"15min":22.541},"min":{"1min":7.2590000000000003,"5min":6.9790000000000001,"15min":6.6609999999999996},"max":{"1min":24.433,"5min":143.75800000000001,"15min":286.96699999999998},"last":12.545},{"interface":"front","average":{"1min":18.093,"5min":23.826000000000001,"15min":26.565000000000001},"min":{"1min":12.166,"5min":12.166,"15min":8.609},"max":{"1min":26.777000000000001,"5min":144.934,"15min":276.17200000000003},"last":17.219999999999999}]},{"osd":6,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":15.897,"5min":17.413,"15min":21.896000000000001},"min":{"1min":7.2969999999999997,"5min":6.5190000000000001,"15min":6.5190000000000001},"max":{"1min":77.432000000000002,"5min":77.432000000000002,"15min":280.36000000000001},"last":18.157},{"interface":"front","average":{"1min":16.088000000000001,"5min":21.170999999999999,"15min":26.117000000000001},"min":{"1min":9.4410000000000007,"5min":7.4020000000000001,"15min":7.4020000000000001},"max":{"1min":25.859999999999999,"5min":109.157,"15min":346.47000000000003},"last":14.581}]},{"osd":7,"last update":"Mon Oct 30 17:54:08 2023","interfaces":[{"interface":"back","average":{"1min":17.742999999999999,"5min":21.178000000000001,"15min":28.771999999999998},"min":{"1min":8.8689999999999998,"5min":8.8689999999999998,"15min":7.6520000000000001},"max":{"1min":54.5,"5min":110.94799999999999,"15min":332.99099999999999},"last":15.162000000000001},{"interface":"front","average":{"1min":20.643000000000001,"5min":22.628,"15min":29.914999999999999},"min":{"1min":11.954000000000001,"5min":10.728,"15min":9.6750000000000007},"max":{"1min":72.408000000000001,"5min":145.00899999999999,"15min":332.64800000000002},"last":13.265000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T17:55:11.388 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.388 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.389 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.389 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.389 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.390 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.390 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.390 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.391 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.391 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.391 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.392 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T17:32:32.679287+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=32, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.392 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.392 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T17:32:32.679287+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=32, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.393 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.393 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T17:32:32.679287+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=32, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.393 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-30T17:32:32.679287+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=32, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.393 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.394 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T17:32:32.679287+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=32, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.394 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.394 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-30T17:32:32.679287+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=32, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.395 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.395 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.395 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-30T17:32:32.679287+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=32, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.396 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.396 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.396 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.397 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.397 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.397 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.398 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.398 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.398 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.399 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.399 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.399 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:11.399 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T17:55:14.450 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:58.416 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:14.456 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:58.422 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:14.471 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:58.437 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:14.503 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:58.443 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:14.505 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:58.444 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:14.517 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:58.456 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:14.526 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:58.492 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:14.527 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:58.493 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:14.540 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:58.506 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:15.127 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:59.066 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:15.128 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:59.067 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:15.135 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:59.075 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:15.220 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:59.186 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:15.221 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:59.188 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:15.230 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:23:59.196 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:15.279 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:59.245 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:15.280 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:59.246 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:15.288 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:23:59.254 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:23.418 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:07.383 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:23.421 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:07.387 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:23.438 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:07.404 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:23.470 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:07.410 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:23.473 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:07.413 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:23.478 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:07.443 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:23.478 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:07.444 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:23.488 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:07.427 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:23.495 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:07.461 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:24.500 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:08.466 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:24.503 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:08.469 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:24.514 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:08.480 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:24.549 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:08.488 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:24.551 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:08.490 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:24.560 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:08.499 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:24.572 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:08.538 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:24.576 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:08.542 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:24.587 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:08.553 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:29.511 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:13.477 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:29.519 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:13.482 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:29.527 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:13.494 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:29.563 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:13.529 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:29.564 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:13.503 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:29.566 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:13.505 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:29.566 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:13.532 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:29.577 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:13.543 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:29.577 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:13.516 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:31.401 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T17:55:31.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.499+0000 7fc413e64640 10 monclient: get_monmap_and_config 2023-10-30T17:55:31.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.499+0000 7fc413e64640 10 monclient: build_initial_monmap 2023-10-30T17:55:31.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.499+0000 7fc413e64640 10 monclient: monmap: 2023-10-30T17:55:31.501 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:31.501 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:31.501 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:31.500121+0000 2023-10-30T17:55:31.501 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:31.500121+0000 2023-10-30T17:55:31.502 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:31.502 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:31.502 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:31.502 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:31.502 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:31.503 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:31.503 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:31.503 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:31.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.499+0000 7fc413e64640 1 Processor -- start 2023-10-30T17:55:31.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 1 -- start start 2023-10-30T17:55:31.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 10 monclient: init 2023-10-30T17:55:31.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:31.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-10-30T17:55:31.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc40c0a8720 0x7fc40c13ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:31.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 10 monclient(hunting): picked mon.noname-a con 0x7fc40c0a8720 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:31.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc40c13f4d0 0x7fc40c1454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:31.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 10 monclient(hunting): picked mon.noname-c con 0x7fc40c13f4d0 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:31.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc40c0a8b00 addr v1:172.21.15.170:6789/0 2023-10-30T17:55:31.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40c0a9fd0 con 0x7fc40c0a8b00 2023-10-30T17:55:31.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:31.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40c0aa240 con 0x7fc40c0a8720 2023-10-30T17:55:31.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:31.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40c0aa520 con 0x7fc40c13f4d0 2023-10-30T17:55:31.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:31.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.500+0000 7fc413e64640 10 monclient(hunting): authenticate will time out at 2713.823905s 2023-10-30T17:55:31.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.501+0000 7fc4113ff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc40c13f4d0 0x7fc40c1454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:31.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.501+0000 7fc412401640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fc40c0a8b00 0x7fc40c141b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:59910/0 (socket says 172.21.15.170:59910) 2023-10-30T17:55:31.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.501+0000 7fc412401640 1 -- 172.21.15.170:0/680588977 learned_addr learned my addr 172.21.15.170:0/680588977 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-30T17:55:31.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.501+0000 7fc411c00640 1 --2- 172.21.15.170:0/680588977 >> v2:172.21.15.170:3300/0 conn(0x7fc40c0a8720 0x7fc40c13ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:31.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.501+0000 7fc4113ff640 10 monclient(hunting): get_auth_request con 0x7fc40c13f4d0 auth_method 0 2023-10-30T17:55:31.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.501+0000 7fc4113ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:31.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.501+0000 7fc4113ff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:31.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.501+0000 7fc4113ff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:31.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.502+0000 7fc4113ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:31.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.502+0000 7fc4113ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:31.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.502+0000 7fc4113ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:31.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.502+0000 7fc411c00640 10 monclient(hunting): get_auth_request con 0x7fc40c0a8720 auth_method 0 2023-10-30T17:55:31.511 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.502+0000 7fc411c00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:31.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.502+0000 7fc411c00640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:31.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.502+0000 7fc411c00640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:31.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.504+0000 7fc4113ff640 10 monclient(hunting): handle_auth_done global_id 21529 payload 274 2023-10-30T17:55:31.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.504+0000 7fc4113ff640 1 -- 172.21.15.170:0/680588977 >> v2:172.21.15.170:3300/0 conn(0x7fc40c0a8720 msgr2=0x7fc40c13ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:31.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.504+0000 7fc4113ff640 1 --2- 172.21.15.170:0/680588977 >> v2:172.21.15.170:3300/0 conn(0x7fc40c0a8720 0x7fc40c13ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.504+0000 7fc4113ff640 1 -- 172.21.15.170:0/680588977 >> v1:172.21.15.170:6789/0 conn(0x7fc40c0a8b00 legacy=0x7fc40c141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:31.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.504+0000 7fc4113ff640 10 monclient: _finish_hunting 0 2023-10-30T17:55:31.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.504+0000 7fc4113ff640 1 monclient: found mon.noname-c 2023-10-30T17:55:31.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.504+0000 7fc4113ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:31.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.504+0000 7fc4113ff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2023-10-30T17:55:31.513 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.504+0000 7fc4113ff640 1 -- 172.21.15.170:0/680588977 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c0b1c80 con 0x7fc40c13f4d0 2023-10-30T17:55:31.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.506+0000 7fc4113ff640 1 --2- 172.21.15.170:0/680588977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c13f4d0 0x7fc40c1454e0 secure :-1 s=READY pgs=5292 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0673a0 tx=0x7fc40802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d440c91281c315c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:31.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 1 -- 172.21.15.170:0/680588977 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408031830 con 0x7fc40c13f4d0 2023-10-30T17:55:31.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:31.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T17:55:31.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 10 monclient: dump: 2023-10-30T17:55:31.518 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:31.519 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:31.519 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:31.519 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:31.519 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:31.519 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:31.520 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:31.520 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:31.520 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:31.520 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:31.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 10 monclient: _finish_auth 0 2023-10-30T17:55:31.520 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 10 monclient: _check_auth_tickets 2023-10-30T17:55:31.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:31.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 1 -- 172.21.15.170:0/680588977 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc408031990 con 0x7fc40c13f4d0 2023-10-30T17:55:31.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc413e64640 5 monclient: authenticate success, global_id 21529 2023-10-30T17:55:31.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc413e64640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:31.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:31.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc413e64640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:31.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc413e64640 4 set_mon_vals no callback set 2023-10-30T17:55:31.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 1 -- 172.21.15.170:0/680588977 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408031d20 con 0x7fc40c13f4d0 2023-10-30T17:55:31.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:31.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:31.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.509+0000 7fc410bfe640 10 monclient: dump: 2023-10-30T17:55:31.523 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:31.523 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:31.523 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:31.523 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:31.523 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:31.524 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:31.524 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:31.524 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:31.524 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:31.524 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:31.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.510+0000 7fc413e64640 10 monclient: shutdown 2023-10-30T17:55:31.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.510+0000 7fc413e64640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:31.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.510+0000 7fc413e64640 1 -- 172.21.15.170:0/680588977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c13f4d0 msgr2=0x7fc40c1454e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:31.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.510+0000 7fc413e64640 1 --2- 172.21.15.170:0/680588977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c13f4d0 0x7fc40c1454e0 secure :-1 s=READY pgs=5292 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0673a0 tx=0x7fc40802b8d0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.510+0000 7fc413e64640 1 -- 172.21.15.170:0/680588977 shutdown_connections 2023-10-30T17:55:31.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.510+0000 7fc413e64640 1 --2- 172.21.15.170:0/680588977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c13f4d0 0x7fc40c1454e0 unknown :-1 s=CLOSED pgs=5292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.510+0000 7fc413e64640 1 --2- 172.21.15.170:0/680588977 >> v2:172.21.15.170:3300/0 conn(0x7fc40c0a8720 0x7fc40c13ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.510+0000 7fc413e64640 1 -- 172.21.15.170:0/680588977 >> 172.21.15.170:0/680588977 conn(0x7fc40c0b7300 msgr2=0x7fc40c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:31.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.510+0000 7fc413e64640 1 -- 172.21.15.170:0/680588977 shutdown_connections 2023-10-30T17:55:31.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.510+0000 7fc413e64640 1 -- 172.21.15.170:0/680588977 wait complete. 2023-10-30T17:55:31.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.511+0000 7fc413e64640 10 monclient: build_initial_monmap 2023-10-30T17:55:31.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.511+0000 7fc413e64640 10 monclient: monmap: 2023-10-30T17:55:31.527 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:31.527 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:31.527 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:31.527 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:31.528 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:31.528 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:31.528 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:31.528 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:31.528 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:31.528 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:31.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.511+0000 7fc413e64640 1 librados: starting msgr at 2023-10-30T17:55:31.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.511+0000 7fc413e64640 1 librados: starting objecter 2023-10-30T17:55:31.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.511+0000 7fc413e64640 1 Processor -- start 2023-10-30T17:55:31.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.511+0000 7fc413e64640 1 -- start start 2023-10-30T17:55:31.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.511+0000 7fc413e64640 1 librados: setting wanted keys 2023-10-30T17:55:31.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.511+0000 7fc413e64640 1 librados: calling monclient init 2023-10-30T17:55:31.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.511+0000 7fc413e64640 10 monclient: init 2023-10-30T17:55:31.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:31.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:55:31.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c0a8720 0x7fc40c0aae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:31.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 10 monclient(hunting): picked mon.noname-b con 0x7fc40c0a8720 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:31.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc40c0a8b00 0x7fc40c0b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:31.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 10 monclient(hunting): picked mon.noname-a con 0x7fc40c0a8b00 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:31.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc40c13f4d0 0x7fc40c144a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:31.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 10 monclient(hunting): picked mon.noname-c con 0x7fc40c13f4d0 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:31.531 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:31.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c0ab450 con 0x7fc40c0a8b00 2023-10-30T17:55:31.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:31.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c1d6b40 con 0x7fc40c0a8720 2023-10-30T17:55:31.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:31.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c1d6cb0 con 0x7fc40c13f4d0 2023-10-30T17:55:31.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:31.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc413e64640 10 monclient(hunting): authenticate will time out at 2713.835520s 2023-10-30T17:55:31.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc411c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c0a8720 0x7fc40c0aae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:31.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc4113ff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc40c0a8b00 0x7fc40c0b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:31.533 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.512+0000 7fc412401640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc40c13f4d0 0x7fc40c144a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:31.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc411c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c0a8720 0x7fc40c0aae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.170:58312/0 (socket says 172.21.15.170:58312) 2023-10-30T17:55:31.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc411c00640 1 -- 172.21.15.170:0/3682593278 learned_addr learned my addr 172.21.15.170:0/3682593278 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:31.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc411c00640 10 monclient(hunting): get_auth_request con 0x7fc40c0a8720 auth_method 0 2023-10-30T17:55:31.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc411c00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:31.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc411c00640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:31.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc411c00640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:31.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc4113ff640 10 monclient(hunting): get_auth_request con 0x7fc40c0a8b00 auth_method 0 2023-10-30T17:55:31.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc4113ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:31.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc4113ff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:31.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc4113ff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:31.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc412401640 10 monclient(hunting): get_auth_request con 0x7fc40c13f4d0 auth_method 0 2023-10-30T17:55:31.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc412401640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:31.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc412401640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:31.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc412401640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:31.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc411c00640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:31.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc411c00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:31.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc411c00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:31.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc4113ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:31.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc4113ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:31.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.513+0000 7fc4113ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:31.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.514+0000 7fc412401640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:31.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.514+0000 7fc412401640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:31.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.514+0000 7fc412401640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:31.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.516+0000 7fc411c00640 10 monclient(hunting): handle_auth_done global_id 21532 payload 824 2023-10-30T17:55:31.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.516+0000 7fc411c00640 1 -- 172.21.15.170:0/3682593278 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc40c13f4d0 msgr2=0x7fc40c144a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:31.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.516+0000 7fc411c00640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc40c13f4d0 0x7fc40c144a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.516+0000 7fc411c00640 1 -- 172.21.15.170:0/3682593278 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc40c0a8b00 msgr2=0x7fc40c0b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:31.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.516+0000 7fc411c00640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc40c0a8b00 0x7fc40c0b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.516+0000 7fc411c00640 10 monclient: _finish_hunting 0 2023-10-30T17:55:31.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.516+0000 7fc411c00640 1 monclient: found mon.noname-b 2023-10-30T17:55:31.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.516+0000 7fc411c00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:31.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.516+0000 7fc411c00640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2023-10-30T17:55:31.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.516+0000 7fc411c00640 1 -- 172.21.15.170:0/3682593278 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc408009400 con 0x7fc40c0a8720 2023-10-30T17:55:31.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.518+0000 7fc411c00640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c0a8720 0x7fc40c0aae30 secure :-1 s=READY pgs=5293 cs=0 l=1 rev1=1 crypto rx=0x7fc40000d800 tx=0x7fc40000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:31.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.518+0000 7fc3fe7fc640 10 client.?.objecter ms_handle_connect 0x7fc40c0a8720 2023-10-30T17:55:31.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.518+0000 7fc3fe7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:31.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 1 -- 172.21.15.170:0/3682593278 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40000b6c0 con 0x7fc40c0a8720 2023-10-30T17:55:31.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:31.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T17:55:31.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 10 monclient: dump: 2023-10-30T17:55:31.542 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:31.542 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:31.543 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:31.543 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:31.543 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:31.543 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:31.543 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:31.544 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:31.544 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:31.544 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:31.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 10 monclient: _finish_auth 0 2023-10-30T17:55:31.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 10 monclient: _check_auth_tickets 2023-10-30T17:55:31.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:31.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 1 -- 172.21.15.170:0/3682593278 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40000b820 con 0x7fc40c0a8720 2023-10-30T17:55:31.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:31.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 1 -- 172.21.15.170:0/3682593278 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40000bbb0 con 0x7fc40c0a8720 2023-10-30T17:55:31.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc413e64640 5 monclient: authenticate success, global_id 21532 2023-10-30T17:55:31.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc413e64640 10 monclient: _renew_subs 2023-10-30T17:55:31.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc413e64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:31.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc413e64640 1 -- 172.21.15.170:0/3682593278 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40c1d6f40 con 0x7fc40c0a8720 2023-10-30T17:55:31.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:31.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T17:55:31.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fe7fc640 10 monclient: dump: 2023-10-30T17:55:31.547 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:31.547 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:31.547 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:31.547 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:31.547 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:31.548 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:31.548 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:31.548 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:31.548 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:31.548 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:31.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc3fffff640 4 set_mon_vals no callback set 2023-10-30T17:55:31.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc413e64640 10 client.21532.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:31.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc413e64640 10 monclient: _renew_subs 2023-10-30T17:55:31.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc413e64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:31.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc413e64640 1 -- 172.21.15.170:0/3682593278 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40c1d7420 con 0x7fc40c0a8720 2023-10-30T17:55:31.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.521+0000 7fc413e64640 1 librados: init done 2023-10-30T17:55:31.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.522+0000 7fc413e64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:31.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.522+0000 7fc413e64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:31.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.522+0000 7fc413e64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2023-10-30T17:55:31.550 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.522+0000 7fc413e64640 1 -- 172.21.15.170:0/3682593278 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40c0ab5c0 con 0x7fc40c0a8720 2023-10-30T17:55:31.551 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.541+0000 7fc3fe7fc640 1 -- 172.21.15.170:0/3682593278 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc40000bd10 con 0x7fc40c0a8720 2023-10-30T17:55:31.551 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.542+0000 7fc3fe7fc640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc3f006c880 0x7fc3f006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:31.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.542+0000 7fc4113ff640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc3f006c880 0x7fc3f006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:31.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.542+0000 7fc4113ff640 10 monclient: get_auth_request con 0x7fc3f006c880 auth_method 0 2023-10-30T17:55:31.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.542+0000 7fc4113ff640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc3f006c880 0x7fc3f006ed40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc408036000 tx=0x7fc40803c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:31.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.542+0000 7fc3fe7fc640 10 client.21532.objecter ms_handle_connect 0x7fc3f006c880 2023-10-30T17:55:31.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.543+0000 7fc3fe7fc640 1 -- 172.21.15.170:0/3682593278 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(914..914 src has 737..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fc4000592f0 con 0x7fc40c0a8720 2023-10-30T17:55:31.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.543+0000 7fc3fe7fc640 10 client.21532.objecter ms_dispatch 0x7fc40c000b90 osd_map(914..914 src has 737..914) v4 2023-10-30T17:55:31.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.544+0000 7fc3fe7fc640 3 client.21532.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:31.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.544+0000 7fc3fe7fc640 3 client.21532.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:31.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.544+0000 7fc3fe7fc640 20 client.21532.objecter dump_active .. 0 homeless 2023-10-30T17:55:31.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.775+0000 7fc3fe7fc640 1 -- 172.21.15.170:0/3682593278 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc400055630 con 0x7fc40c0a8720 2023-10-30T17:55:31.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.775+0000 7fc3fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:31.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.775+0000 7fc3fe7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:31.942 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.941+0000 7fc413e64640 1 -- 172.21.15.170:0/3682593278 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc40c145a20 con 0x7fc3f006c880 2023-10-30T17:55:31.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.946+0000 7fc3fe7fc640 1 -- 172.21.15.170:0/3682593278 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151999 (secure 0 0 0) 0x7fc408044400 con 0x7fc3f006c880 2023-10-30T17:55:31.949 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:55:31.949 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-10-30T17:55:31.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.948+0000 7fc413e64640 10 librados: watch_flush enter 2023-10-30T17:55:31.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.948+0000 7fc413e64640 10 librados: watch_flush exit 2023-10-30T17:55:31.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.948+0000 7fc413e64640 20 client.21532.objecter shutdown clearing up homeless session... 2023-10-30T17:55:31.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.948+0000 7fc413e64640 10 client.21532.objecter successfully canceled tick 2023-10-30T17:55:31.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 -- 172.21.15.170:0/3682593278 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc3f006c880 msgr2=0x7fc3f006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:31.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc3f006c880 0x7fc3f006ed40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc408036000 tx=0x7fc40803c040 comp rx=0 tx=0).stop 2023-10-30T17:55:31.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 10 monclient: shutdown 2023-10-30T17:55:31.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:31.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 -- 172.21.15.170:0/3682593278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c0a8720 msgr2=0x7fc40c0aae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:31.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c0a8720 0x7fc40c0aae30 secure :-1 s=READY pgs=5293 cs=0 l=1 rev1=1 crypto rx=0x7fc40000d800 tx=0x7fc40000dcc0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 -- 172.21.15.170:0/3682593278 shutdown_connections 2023-10-30T17:55:31.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7fc3f006c880 0x7fc3f006ed40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc40c13f4d0 0x7fc40c144a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc40c0a8b00 0x7fc40c0b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 --2- 172.21.15.170:0/3682593278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc40c0a8720 0x7fc40c0aae30 unknown :-1 s=CLOSED pgs=5293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:31.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 -- 172.21.15.170:0/3682593278 >> 172.21.15.170:0/3682593278 conn(0x7fc40c0b7300 msgr2=0x7fc40c073960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:31.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 -- 172.21.15.170:0/3682593278 shutdown_connections 2023-10-30T17:55:31.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 -- 172.21.15.170:0/3682593278 wait complete. 2023-10-30T17:55:31.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:31.949+0000 7fc413e64640 1 librados: shutdown 2023-10-30T17:55:31.962 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1610,"stamp":"2023-10-30T17:55:31.332430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":568568,"kb_used_data":60984,"kb_used_omap":12,"kb_used_meta":507507,"kb_avail":754406152,"statfs":{"total":773094113280,"available":772511899648,"internally_reserved":0,"allocated":62447616,"data_stored":34691458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":519687736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002729"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:09.591816+0000","last_change":"2023-10-30T17:55:09.591816+0000","last_active":"2023-10-30T17:55:09.591816+0000","last_peered":"2023-10-30T17:55:09.591816+0000","last_clean":"2023-10-30T17:55:09.591816+0000","last_became_active":"2023-10-30T17:33:37.643213+0000","last_became_peered":"2023-10-30T17:33:37.643213+0000","last_unstale":"2023-10-30T17:55:09.591816+0000","last_undegraded":"2023-10-30T17:55:09.591816+0000","last_fullsized":"2023-10-30T17:55:09.591816+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:09.589496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:09.589496+0000","last_clean_scrub_stamp":"2023-10-30T17:55:09.589496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:04:42.753690+0000","scrub_duration":0.046746827999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:11.164191+0000","last_change":"2023-10-30T17:55:11.164191+0000","last_active":"2023-10-30T17:55:11.164191+0000","last_peered":"2023-10-30T17:55:11.164191+0000","last_clean":"2023-10-30T17:55:11.164191+0000","last_became_active":"2023-10-30T17:33:37.827806+0000","last_became_peered":"2023-10-30T17:33:37.827806+0000","last_unstale":"2023-10-30T17:55:11.164191+0000","last_undegraded":"2023-10-30T17:55:11.164191+0000","last_fullsized":"2023-10-30T17:55:11.164191+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:11.161754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:11.161754+0000","last_clean_scrub_stamp":"2023-10-30T17:55:11.161754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:38.406882+0000","scrub_duration":0.050985619000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:23.765774+0000","last_change":"2023-10-30T17:55:23.767753+0000","last_active":"2023-10-30T17:55:23.765774+0000","last_peered":"2023-10-30T17:55:23.765774+0000","last_clean":"2023-10-30T17:55:23.765774+0000","last_became_active":"2023-10-30T17:33:40.511287+0000","last_became_peered":"2023-10-30T17:33:40.511287+0000","last_unstale":"2023-10-30T17:55:23.765774+0000","last_undegraded":"2023-10-30T17:55:23.765774+0000","last_fullsized":"2023-10-30T17:55:23.765774+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:23.763969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:23.763969+0000","last_clean_scrub_stamp":"2023-10-30T17:55:23.763969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:40:10.265375+0000","scrub_duration":0.099663560999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":863,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:15.166708+0000","last_change":"2023-10-30T17:55:15.170251+0000","last_active":"2023-10-30T17:55:15.166708+0000","last_peered":"2023-10-30T17:55:15.166708+0000","last_clean":"2023-10-30T17:55:15.166708+0000","last_became_active":"2023-10-30T17:33:38.581457+0000","last_became_peered":"2023-10-30T17:33:38.581457+0000","last_unstale":"2023-10-30T17:55:15.166708+0000","last_undegraded":"2023-10-30T17:55:15.166708+0000","last_fullsized":"2023-10-30T17:55:15.166708+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:15.165342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:15.165342+0000","last_clean_scrub_stamp":"2023-10-30T17:55:15.165342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:50.836306+0000","scrub_duration":0.048933965000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":870,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:16.497473+0000","last_change":"2023-10-30T17:55:16.498710+0000","last_active":"2023-10-30T17:55:16.497473+0000","last_peered":"2023-10-30T17:55:16.497473+0000","last_clean":"2023-10-30T17:55:16.497473+0000","last_became_active":"2023-10-30T17:33:46.629442+0000","last_became_peered":"2023-10-30T17:33:46.629442+0000","last_unstale":"2023-10-30T17:55:16.497473+0000","last_undegraded":"2023-10-30T17:55:16.497473+0000","last_fullsized":"2023-10-30T17:55:16.497473+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:16.496140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:16.496140+0000","last_clean_scrub_stamp":"2023-10-30T17:55:16.496140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:16.307896+0000","scrub_duration":0.044569352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:24.718152+0000","last_change":"2023-10-30T17:55:24.719229+0000","last_active":"2023-10-30T17:55:24.718152+0000","last_peered":"2023-10-30T17:55:24.718152+0000","last_clean":"2023-10-30T17:55:24.718152+0000","last_became_active":"2023-10-30T17:33:39.135750+0000","last_became_peered":"2023-10-30T17:33:39.135750+0000","last_unstale":"2023-10-30T17:55:24.718152+0000","last_undegraded":"2023-10-30T17:55:24.718152+0000","last_fullsized":"2023-10-30T17:55:24.718152+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:24.716758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:24.716758+0000","last_clean_scrub_stamp":"2023-10-30T17:55:24.716758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:48.459846+0000","scrub_duration":0.047575569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:25.744263+0000","last_change":"2023-10-30T17:55:25.748963+0000","last_active":"2023-10-30T17:55:25.744263+0000","last_peered":"2023-10-30T17:55:25.744263+0000","last_clean":"2023-10-30T17:55:25.744263+0000","last_became_active":"2023-10-30T17:33:51.001524+0000","last_became_peered":"2023-10-30T17:33:51.001524+0000","last_unstale":"2023-10-30T17:55:25.744263+0000","last_undegraded":"2023-10-30T17:55:25.744263+0000","last_fullsized":"2023-10-30T17:55:25.744263+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:25.742821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:25.742821+0000","last_clean_scrub_stamp":"2023-10-30T17:55:25.742821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:29.684201+0000","scrub_duration":0.071310417000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":879,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:18.692949+0000","last_change":"2023-10-30T17:55:18.694788+0000","last_active":"2023-10-30T17:55:18.692949+0000","last_peered":"2023-10-30T17:55:18.692949+0000","last_clean":"2023-10-30T17:55:18.692949+0000","last_became_active":"2023-10-30T17:33:58.874661+0000","last_became_peered":"2023-10-30T17:33:58.874661+0000","last_unstale":"2023-10-30T17:55:18.692949+0000","last_undegraded":"2023-10-30T17:55:18.692949+0000","last_fullsized":"2023-10-30T17:55:18.692949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:18.691215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:18.691215+0000","last_clean_scrub_stamp":"2023-10-30T17:55:18.691215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:51:38.631964+0000","scrub_duration":0.040631130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":860,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:27.285688+0000","last_change":"2023-10-30T17:55:27.286706+0000","last_active":"2023-10-30T17:55:27.285688+0000","last_peered":"2023-10-30T17:55:27.285688+0000","last_clean":"2023-10-30T17:55:27.285688+0000","last_became_active":"2023-10-30T17:33:39.086390+0000","last_became_peered":"2023-10-30T17:33:39.086390+0000","last_unstale":"2023-10-30T17:55:27.285688+0000","last_undegraded":"2023-10-30T17:55:27.285688+0000","last_fullsized":"2023-10-30T17:55:27.285688+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:27.284380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:27.284380+0000","last_clean_scrub_stamp":"2023-10-30T17:55:27.284380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:00:33.325612+0000","scrub_duration":0.046134963000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:17.576975+0000","last_change":"2023-10-30T17:55:17.578046+0000","last_active":"2023-10-30T17:55:17.576975+0000","last_peered":"2023-10-30T17:55:17.576975+0000","last_clean":"2023-10-30T17:55:17.576975+0000","last_became_active":"2023-10-30T17:33:39.965244+0000","last_became_peered":"2023-10-30T17:33:39.965244+0000","last_unstale":"2023-10-30T17:55:17.576975+0000","last_undegraded":"2023-10-30T17:55:17.576975+0000","last_fullsized":"2023-10-30T17:55:17.576975+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:17.575634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:17.575634+0000","last_clean_scrub_stamp":"2023-10-30T17:55:17.575634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:48.273182+0000","scrub_duration":0.042278923000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:23.456536+0000","last_change":"2023-10-30T17:55:23.457600+0000","last_active":"2023-10-30T17:55:23.456536+0000","last_peered":"2023-10-30T17:55:23.456536+0000","last_clean":"2023-10-30T17:55:23.456536+0000","last_became_active":"2023-10-30T17:33:46.562080+0000","last_became_peered":"2023-10-30T17:33:46.562080+0000","last_unstale":"2023-10-30T17:55:23.456536+0000","last_undegraded":"2023-10-30T17:55:23.456536+0000","last_fullsized":"2023-10-30T17:55:23.456536+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:23.455039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:23.455039+0000","last_clean_scrub_stamp":"2023-10-30T17:55:23.455039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:26:27.125563+0000","scrub_duration":0.072097618000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":879,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:12.512329+0000","last_change":"2023-10-30T17:55:12.513855+0000","last_active":"2023-10-30T17:55:12.512329+0000","last_peered":"2023-10-30T17:55:12.512329+0000","last_clean":"2023-10-30T17:55:12.512329+0000","last_became_active":"2023-10-30T17:33:40.798989+0000","last_became_peered":"2023-10-30T17:33:40.798989+0000","last_unstale":"2023-10-30T17:55:12.512329+0000","last_undegraded":"2023-10-30T17:55:12.512329+0000","last_fullsized":"2023-10-30T17:55:12.512329+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:12.510278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:12.510278+0000","last_clean_scrub_stamp":"2023-10-30T17:55:12.510278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:35.159349+0000","scrub_duration":0.041716329000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:21.357369+0000","last_change":"2023-10-30T17:55:21.358376+0000","last_active":"2023-10-30T17:55:21.357369+0000","last_peered":"2023-10-30T17:55:21.357369+0000","last_clean":"2023-10-30T17:55:21.357369+0000","last_became_active":"2023-10-30T17:32:35.367487+0000","last_became_peered":"2023-10-30T17:32:35.367487+0000","last_unstale":"2023-10-30T17:55:21.357369+0000","last_undegraded":"2023-10-30T17:55:21.357369+0000","last_fullsized":"2023-10-30T17:55:21.357369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:21.356030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:21.356030+0000","last_clean_scrub_stamp":"2023-10-30T17:55:21.356030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:04.756228+0000","scrub_duration":0.033632153999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.279583+0000","last_change":"2023-10-30T17:33:38.291772+0000","last_active":"2023-10-30T17:55:04.279583+0000","last_peered":"2023-10-30T17:55:04.279583+0000","last_clean":"2023-10-30T17:55:04.279583+0000","last_became_active":"2023-10-30T17:33:38.284317+0000","last_became_peered":"2023-10-30T17:33:38.284317+0000","last_unstale":"2023-10-30T17:55:04.279583+0000","last_undegraded":"2023-10-30T17:55:04.279583+0000","last_fullsized":"2023-10-30T17:55:04.279583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":877,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:08.573133+0000","last_change":"2023-10-30T17:55:08.574169+0000","last_active":"2023-10-30T17:55:08.573133+0000","last_peered":"2023-10-30T17:55:08.573133+0000","last_clean":"2023-10-30T17:55:08.573133+0000","last_became_active":"2023-10-30T17:32:34.557021+0000","last_became_peered":"2023-10-30T17:32:34.557021+0000","last_unstale":"2023-10-30T17:55:08.573133+0000","last_undegraded":"2023-10-30T17:55:08.573133+0000","last_fullsized":"2023-10-30T17:55:08.573133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:08.571819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:08.571819+0000","last_clean_scrub_stamp":"2023-10-30T17:55:08.571819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:38:46.258857+0000","scrub_duration":0.028442723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"914'8","reported_seq":875,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:28.321727+0000","last_change":"2023-10-30T17:55:28.323254+0000","last_active":"2023-10-30T17:55:28.321727+0000","last_peered":"2023-10-30T17:55:28.321727+0000","last_clean":"2023-10-30T17:55:28.321727+0000","last_became_active":"2023-10-30T17:33:39.211129+0000","last_became_peered":"2023-10-30T17:33:39.211129+0000","last_unstale":"2023-10-30T17:55:28.321727+0000","last_undegraded":"2023-10-30T17:55:28.321727+0000","last_fullsized":"2023-10-30T17:55:28.321727+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'8","last_scrub_stamp":"2023-10-30T17:55:28.320436+0000","last_deep_scrub":"914'8","last_deep_scrub_stamp":"2023-10-30T17:55:28.320436+0000","last_clean_scrub_stamp":"2023-10-30T17:55:28.320436+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:58.798076+0000","scrub_duration":0.10408546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:19.390591+0000","last_change":"2023-10-30T17:55:19.391631+0000","last_active":"2023-10-30T17:55:19.390591+0000","last_peered":"2023-10-30T17:55:19.390591+0000","last_clean":"2023-10-30T17:55:19.390591+0000","last_became_active":"2023-10-30T17:32:35.357489+0000","last_became_peered":"2023-10-30T17:32:35.357489+0000","last_unstale":"2023-10-30T17:55:19.390591+0000","last_undegraded":"2023-10-30T17:55:19.390591+0000","last_fullsized":"2023-10-30T17:55:19.390591+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:19.389304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:19.389304+0000","last_clean_scrub_stamp":"2023-10-30T17:55:19.389304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:18.913618+0000","scrub_duration":0.030571640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":869,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:06.175371+0000","last_change":"2023-10-30T17:55:06.176783+0000","last_active":"2023-10-30T17:55:06.175371+0000","last_peered":"2023-10-30T17:55:06.175371+0000","last_clean":"2023-10-30T17:55:06.175371+0000","last_became_active":"2023-10-30T17:33:40.584323+0000","last_became_peered":"2023-10-30T17:33:40.584323+0000","last_unstale":"2023-10-30T17:55:06.175371+0000","last_undegraded":"2023-10-30T17:55:06.175371+0000","last_fullsized":"2023-10-30T17:55:06.175371+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:06.173670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:06.173670+0000","last_clean_scrub_stamp":"2023-10-30T17:55:06.173670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:41.569418+0000","scrub_duration":0.033937040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":873,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:11.579114+0000","last_change":"2023-10-30T17:55:11.580393+0000","last_active":"2023-10-30T17:55:11.579114+0000","last_peered":"2023-10-30T17:55:11.579114+0000","last_clean":"2023-10-30T17:55:11.579114+0000","last_became_active":"2023-10-30T17:32:34.631494+0000","last_became_peered":"2023-10-30T17:32:34.631494+0000","last_unstale":"2023-10-30T17:55:11.579114+0000","last_undegraded":"2023-10-30T17:55:11.579114+0000","last_fullsized":"2023-10-30T17:55:11.579114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:11.576330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:11.576330+0000","last_clean_scrub_stamp":"2023-10-30T17:55:11.576330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:45:39.718092+0000","scrub_duration":0.043490681000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.296018+0000","last_change":"2023-10-30T17:33:46.593988+0000","last_active":"2023-10-30T17:55:04.296018+0000","last_peered":"2023-10-30T17:55:04.296018+0000","last_clean":"2023-10-30T17:55:04.296018+0000","last_became_active":"2023-10-30T17:33:46.588555+0000","last_became_peered":"2023-10-30T17:33:46.588555+0000","last_unstale":"2023-10-30T17:55:04.296018+0000","last_undegraded":"2023-10-30T17:55:04.296018+0000","last_fullsized":"2023-10-30T17:55:04.296018+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:16.403101+0000","last_change":"2023-10-30T17:55:16.404127+0000","last_active":"2023-10-30T17:55:16.403101+0000","last_peered":"2023-10-30T17:55:16.403101+0000","last_clean":"2023-10-30T17:55:16.403101+0000","last_became_active":"2023-10-30T17:32:35.383414+0000","last_became_peered":"2023-10-30T17:32:35.383414+0000","last_unstale":"2023-10-30T17:55:16.403101+0000","last_undegraded":"2023-10-30T17:55:16.403101+0000","last_fullsized":"2023-10-30T17:55:16.403101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:16.401822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:16.401822+0000","last_clean_scrub_stamp":"2023-10-30T17:55:16.401822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:24.597562+0000","scrub_duration":0.028021751000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":885,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:17.420795+0000","last_change":"2023-10-30T17:55:17.421804+0000","last_active":"2023-10-30T17:55:17.420795+0000","last_peered":"2023-10-30T17:55:17.420795+0000","last_clean":"2023-10-30T17:55:17.420795+0000","last_became_active":"2023-10-30T17:33:38.176989+0000","last_became_peered":"2023-10-30T17:33:38.176989+0000","last_unstale":"2023-10-30T17:55:17.420795+0000","last_undegraded":"2023-10-30T17:55:17.420795+0000","last_fullsized":"2023-10-30T17:55:17.420795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:17.419519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:17.419519+0000","last_clean_scrub_stamp":"2023-10-30T17:55:17.419519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:04:02.018126+0000","scrub_duration":0.036305485999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":893,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:16.387462+0000","last_change":"2023-10-30T17:55:16.387462+0000","last_active":"2023-10-30T17:55:16.387462+0000","last_peered":"2023-10-30T17:55:16.387462+0000","last_clean":"2023-10-30T17:55:16.387462+0000","last_became_active":"2023-10-30T17:32:34.287031+0000","last_became_peered":"2023-10-30T17:32:34.287031+0000","last_unstale":"2023-10-30T17:55:16.387462+0000","last_undegraded":"2023-10-30T17:55:16.387462+0000","last_fullsized":"2023-10-30T17:55:16.387462+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:16.385069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:16.385069+0000","last_clean_scrub_stamp":"2023-10-30T17:55:16.385069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:04.642185+0000","scrub_duration":0.043308434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":885,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:21.487482+0000","last_change":"2023-10-30T17:55:21.488487+0000","last_active":"2023-10-30T17:55:21.487482+0000","last_peered":"2023-10-30T17:55:21.487482+0000","last_clean":"2023-10-30T17:55:21.487482+0000","last_became_active":"2023-10-30T17:33:41.116885+0000","last_became_peered":"2023-10-30T17:33:41.116885+0000","last_unstale":"2023-10-30T17:55:21.487482+0000","last_undegraded":"2023-10-30T17:55:21.487482+0000","last_fullsized":"2023-10-30T17:55:21.487482+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:21.486124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:21.486124+0000","last_clean_scrub_stamp":"2023-10-30T17:55:21.486124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:12.382300+0000","scrub_duration":0.040156958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":866,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:09.309575+0000","last_change":"2023-10-30T17:55:09.309575+0000","last_active":"2023-10-30T17:55:09.309575+0000","last_peered":"2023-10-30T17:55:09.309575+0000","last_clean":"2023-10-30T17:55:09.309575+0000","last_became_active":"2023-10-30T17:32:35.277163+0000","last_became_peered":"2023-10-30T17:32:35.277163+0000","last_unstale":"2023-10-30T17:55:09.309575+0000","last_undegraded":"2023-10-30T17:55:09.309575+0000","last_fullsized":"2023-10-30T17:55:09.309575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:09.302612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:09.302612+0000","last_clean_scrub_stamp":"2023-10-30T17:55:09.302612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:08:21.503163+0000","scrub_duration":0.044067580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":852,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:07.516566+0000","last_change":"2023-10-30T17:33:43.745646+0000","last_active":"2023-10-30T17:55:07.516566+0000","last_peered":"2023-10-30T17:55:07.516566+0000","last_clean":"2023-10-30T17:55:07.516566+0000","last_became_active":"2023-10-30T17:33:43.736021+0000","last_became_peered":"2023-10-30T17:33:43.736021+0000","last_unstale":"2023-10-30T17:55:07.516566+0000","last_undegraded":"2023-10-30T17:55:07.516566+0000","last_fullsized":"2023-10-30T17:55:07.516566+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:23.334243+0000","last_change":"2023-10-30T17:55:23.335344+0000","last_active":"2023-10-30T17:55:23.334243+0000","last_peered":"2023-10-30T17:55:23.334243+0000","last_clean":"2023-10-30T17:55:23.334243+0000","last_became_active":"2023-10-30T17:32:35.623395+0000","last_became_peered":"2023-10-30T17:32:35.623395+0000","last_unstale":"2023-10-30T17:55:23.334243+0000","last_undegraded":"2023-10-30T17:55:23.334243+0000","last_fullsized":"2023-10-30T17:55:23.334243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:23.332803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:23.332803+0000","last_clean_scrub_stamp":"2023-10-30T17:55:23.332803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:07:30.217748+0000","scrub_duration":0.035208454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"914'78","reported_seq":978,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:10.218608+0000","last_change":"2023-10-30T17:55:10.220901+0000","last_active":"2023-10-30T17:55:10.218608+0000","last_peered":"2023-10-30T17:55:10.218608+0000","last_clean":"2023-10-30T17:55:10.218608+0000","last_became_active":"2023-10-30T17:32:28.686958+0000","last_became_peered":"2023-10-30T17:32:28.686958+0000","last_unstale":"2023-10-30T17:55:10.218608+0000","last_undegraded":"2023-10-30T17:55:10.218608+0000","last_fullsized":"2023-10-30T17:55:10.218608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'78","last_scrub_stamp":"2023-10-30T17:55:10.216301+0000","last_deep_scrub":"914'78","last_deep_scrub_stamp":"2023-10-30T17:55:10.216301+0000","last_clean_scrub_stamp":"2023-10-30T17:55:10.216301+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:16:00.152248+0000","scrub_duration":0.12796829700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":869,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:07.218625+0000","last_change":"2023-10-30T17:55:07.219619+0000","last_active":"2023-10-30T17:55:07.218625+0000","last_peered":"2023-10-30T17:55:07.218625+0000","last_clean":"2023-10-30T17:55:07.218625+0000","last_became_active":"2023-10-30T17:33:40.602436+0000","last_became_peered":"2023-10-30T17:33:40.602436+0000","last_unstale":"2023-10-30T17:55:07.218625+0000","last_undegraded":"2023-10-30T17:55:07.218625+0000","last_fullsized":"2023-10-30T17:55:07.218625+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:07.217190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:07.217190+0000","last_clean_scrub_stamp":"2023-10-30T17:55:07.217190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:25.972202+0000","scrub_duration":0.033129934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":852,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:04.451334+0000","last_change":"2023-10-30T17:33:38.623594+0000","last_active":"2023-10-30T17:55:04.451334+0000","last_peered":"2023-10-30T17:55:04.451334+0000","last_clean":"2023-10-30T17:55:04.451334+0000","last_became_active":"2023-10-30T17:33:38.616054+0000","last_became_peered":"2023-10-30T17:33:38.616054+0000","last_unstale":"2023-10-30T17:55:04.451334+0000","last_undegraded":"2023-10-30T17:55:04.451334+0000","last_fullsized":"2023-10-30T17:55:04.451334+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:19.186826+0000","last_change":"2023-10-30T17:55:19.187904+0000","last_active":"2023-10-30T17:55:19.186826+0000","last_peered":"2023-10-30T17:55:19.186826+0000","last_clean":"2023-10-30T17:55:19.186826+0000","last_became_active":"2023-10-30T17:33:40.735511+0000","last_became_peered":"2023-10-30T17:33:40.735511+0000","last_unstale":"2023-10-30T17:55:19.186826+0000","last_undegraded":"2023-10-30T17:55:19.186826+0000","last_fullsized":"2023-10-30T17:55:19.186826+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:19.185432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:19.185432+0000","last_clean_scrub_stamp":"2023-10-30T17:55:19.185432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:17.820548+0000","scrub_duration":0.067969350999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":867,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:14.589320+0000","last_change":"2023-10-30T17:55:14.590436+0000","last_active":"2023-10-30T17:55:14.589320+0000","last_peered":"2023-10-30T17:55:14.589320+0000","last_clean":"2023-10-30T17:55:14.589320+0000","last_became_active":"2023-10-30T17:33:39.228561+0000","last_became_peered":"2023-10-30T17:33:39.228561+0000","last_unstale":"2023-10-30T17:55:14.589320+0000","last_undegraded":"2023-10-30T17:55:14.589320+0000","last_fullsized":"2023-10-30T17:55:14.589320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:14.587906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:14.587906+0000","last_clean_scrub_stamp":"2023-10-30T17:55:14.587906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:23:56.357532+0000","scrub_duration":0.056479152999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":884,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:20.480379+0000","last_change":"2023-10-30T17:55:20.481377+0000","last_active":"2023-10-30T17:55:20.480379+0000","last_peered":"2023-10-30T17:55:20.480379+0000","last_clean":"2023-10-30T17:55:20.480379+0000","last_became_active":"2023-10-30T17:33:41.605208+0000","last_became_peered":"2023-10-30T17:33:41.605208+0000","last_unstale":"2023-10-30T17:55:20.480379+0000","last_undegraded":"2023-10-30T17:55:20.480379+0000","last_fullsized":"2023-10-30T17:55:20.480379+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:20.479126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:20.479126+0000","last_clean_scrub_stamp":"2023-10-30T17:55:20.479126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:49:32.198607+0000","scrub_duration":0.066395858000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":880,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:13.534354+0000","last_change":"2023-10-30T17:55:13.534354+0000","last_active":"2023-10-30T17:55:13.534354+0000","last_peered":"2023-10-30T17:55:13.534354+0000","last_clean":"2023-10-30T17:55:13.534354+0000","last_became_active":"2023-10-30T17:33:44.434104+0000","last_became_peered":"2023-10-30T17:33:44.434104+0000","last_unstale":"2023-10-30T17:55:13.534354+0000","last_undegraded":"2023-10-30T17:55:13.534354+0000","last_fullsized":"2023-10-30T17:55:13.534354+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:13.528576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:13.528576+0000","last_clean_scrub_stamp":"2023-10-30T17:55:13.528576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:35:11.000117+0000","scrub_duration":0.031392597000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":868,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:12.587925+0000","last_change":"2023-10-30T17:55:12.589027+0000","last_active":"2023-10-30T17:55:12.587925+0000","last_peered":"2023-10-30T17:55:12.587925+0000","last_clean":"2023-10-30T17:55:12.587925+0000","last_became_active":"2023-10-30T17:33:43.322860+0000","last_became_peered":"2023-10-30T17:33:43.322860+0000","last_unstale":"2023-10-30T17:55:12.587925+0000","last_undegraded":"2023-10-30T17:55:12.587925+0000","last_fullsized":"2023-10-30T17:55:12.587925+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:12.586555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:12.586555+0000","last_clean_scrub_stamp":"2023-10-30T17:55:12.586555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:26.081277+0000","scrub_duration":0.065328111999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":880,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:11.450149+0000","last_change":"2023-10-30T17:55:11.450149+0000","last_active":"2023-10-30T17:55:11.450149+0000","last_peered":"2023-10-30T17:55:11.450149+0000","last_clean":"2023-10-30T17:55:11.450149+0000","last_became_active":"2023-10-30T17:33:53.130615+0000","last_became_peered":"2023-10-30T17:33:53.130615+0000","last_unstale":"2023-10-30T17:55:11.450149+0000","last_undegraded":"2023-10-30T17:55:11.450149+0000","last_fullsized":"2023-10-30T17:55:11.450149+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:11.447787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:11.447787+0000","last_clean_scrub_stamp":"2023-10-30T17:55:11.447787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:15:08.444731+0000","scrub_duration":0.02798376,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:13.182186+0000","last_change":"2023-10-30T17:55:13.182186+0000","last_active":"2023-10-30T17:55:13.182186+0000","last_peered":"2023-10-30T17:55:13.182186+0000","last_clean":"2023-10-30T17:55:13.182186+0000","last_became_active":"2023-10-30T17:33:37.540749+0000","last_became_peered":"2023-10-30T17:33:37.540749+0000","last_unstale":"2023-10-30T17:55:13.182186+0000","last_undegraded":"2023-10-30T17:55:13.182186+0000","last_fullsized":"2023-10-30T17:55:13.182186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:13.179784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:13.179784+0000","last_clean_scrub_stamp":"2023-10-30T17:55:13.179784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:38.127850+0000","scrub_duration":0.071081134000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":852,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:07.522811+0000","last_change":"2023-10-30T17:33:58.392321+0000","last_active":"2023-10-30T17:55:07.522811+0000","last_peered":"2023-10-30T17:55:07.522811+0000","last_clean":"2023-10-30T17:55:07.522811+0000","last_became_active":"2023-10-30T17:33:58.383320+0000","last_became_peered":"2023-10-30T17:33:58.383320+0000","last_unstale":"2023-10-30T17:55:07.522811+0000","last_undegraded":"2023-10-30T17:55:07.522811+0000","last_fullsized":"2023-10-30T17:55:07.522811+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":870,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:19.494465+0000","last_change":"2023-10-30T17:55:19.494465+0000","last_active":"2023-10-30T17:55:19.494465+0000","last_peered":"2023-10-30T17:55:19.494465+0000","last_clean":"2023-10-30T17:55:19.494465+0000","last_became_active":"2023-10-30T17:33:38.589479+0000","last_became_peered":"2023-10-30T17:33:38.589479+0000","last_unstale":"2023-10-30T17:55:19.494465+0000","last_undegraded":"2023-10-30T17:55:19.494465+0000","last_fullsized":"2023-10-30T17:55:19.494465+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:19.489037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:19.489037+0000","last_clean_scrub_stamp":"2023-10-30T17:55:19.489037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:46.197528+0000","scrub_duration":0.068705826999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:21.428804+0000","last_change":"2023-10-30T17:55:21.429918+0000","last_active":"2023-10-30T17:55:21.428804+0000","last_peered":"2023-10-30T17:55:21.428804+0000","last_clean":"2023-10-30T17:55:21.428804+0000","last_became_active":"2023-10-30T17:33:40.476315+0000","last_became_peered":"2023-10-30T17:33:40.476315+0000","last_unstale":"2023-10-30T17:55:21.428804+0000","last_undegraded":"2023-10-30T17:55:21.428804+0000","last_fullsized":"2023-10-30T17:55:21.428804+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:21.427013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:21.427013+0000","last_clean_scrub_stamp":"2023-10-30T17:55:21.427013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:15.296878+0000","scrub_duration":0.036816092000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":852,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:07.519515+0000","last_change":"2023-10-30T17:33:38.803571+0000","last_active":"2023-10-30T17:55:07.519515+0000","last_peered":"2023-10-30T17:55:07.519515+0000","last_clean":"2023-10-30T17:55:07.519515+0000","last_became_active":"2023-10-30T17:33:38.795801+0000","last_became_peered":"2023-10-30T17:33:38.795801+0000","last_unstale":"2023-10-30T17:55:07.519515+0000","last_undegraded":"2023-10-30T17:55:07.519515+0000","last_fullsized":"2023-10-30T17:55:07.519515+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:33:34.187727+0000","last_clean_scrub_stamp":"2023-10-30T17:33:34.187727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":17,"seq":73014444308,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60908,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310932,"statfs":{"total":96636764160,"available":96574394368,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 17:55:23 2023","interfaces":[{"interface":"back","average":{"1min":12.914,"5min":14.653,"15min":19.036999999999999},"min":{"1min":5.3259999999999996,"5min":4.2460000000000004,"15min":4.2460000000000004},"max":{"1min":63.363,"5min":217.97399999999999,"15min":296.49400000000003},"last":6.1740000000000004},{"interface":"front","average":{"1min":10.414,"5min":15.103999999999999,"15min":19.773},"min":{"1min":5.5309999999999997,"5min":5.5309999999999997,"15min":4.6299999999999999},"max":{"1min":25.385000000000002,"5min":134.833,"15min":276.49799999999999},"last":9.391}]},{"osd":2,"last update":"Mon Oct 30 17:55:20 2023","interfaces":[{"interface":"back","average":{"1min":14.949999999999999,"5min":16.766999999999999,"15min":23.184999999999999},"min":{"1min":8.1669999999999998,"5min":6.319,"15min":5.915},"max":{"1min":33.576999999999998,"5min":120.46299999999999,"15min":591.02499999999998},"last":12.914999999999999},{"interface":"front","average":{"1min":15.061,"5min":18.091000000000001,"15min":23.324000000000002},"min":{"1min":9.3659999999999997,"5min":7.6520000000000001,"15min":6.4930000000000003},"max":{"1min":70.671000000000006,"5min":129.03299999999999,"15min":259.52300000000002},"last":11.198}]},{"osd":3,"last update":"Mon Oct 30 17:54:58 2023","interfaces":[{"interface":"back","average":{"1min":13.741,"5min":75.924000000000007,"15min":40.366999999999997},"min":{"1min":6.4290000000000003,"5min":6.4290000000000003,"15min":5.8099999999999996},"max":{"1min":62.359000000000002,"5min":5036.1790000000001,"15min":5036.1790000000001},"last":6.9610000000000003},{"interface":"front","average":{"1min":17.495999999999999,"5min":79.316999999999993,"15min":44.966999999999999},"min":{"1min":7.6950000000000003,"5min":7.2770000000000001,"15min":6.4909999999999997},"max":{"1min":47.174999999999997,"5min":5057.1229999999996,"15min":5057.1229999999996},"last":14.250999999999999}]},{"osd":4,"last update":"Mon Oct 30 17:55:23 2023","interfaces":[{"interface":"back","average":{"1min":15.284000000000001,"5min":19.625,"15min":21.283000000000001},"min":{"1min":7.5380000000000003,"5min":7.3259999999999996,"15min":6.3689999999999998},"max":{"1min":64.394000000000005,"5min":159.94399999999999,"15min":281.16399999999999},"last":20.306000000000001},{"interface":"front","average":{"1min":17.460000000000001,"5min":23.212,"15min":26},"min":{"1min":8.9160000000000004,"5min":8.9160000000000004,"15min":5.8879999999999999},"max":{"1min":49.860999999999997,"5min":198.04900000000001,"15min":218.85300000000001},"last":18.884}]},{"osd":5,"last update":"Mon Oct 30 17:55:23 2023","interfaces":[{"interface":"back","average":{"1min":17.294,"5min":22.064,"15min":26.204000000000001},"min":{"1min":8.1850000000000005,"5min":8.1850000000000005,"15min":7.9370000000000003},"max":{"1min":80.078000000000003,"5min":260.81099999999998,"15min":260.81099999999998},"last":10.035},{"interface":"front","average":{"1min":18.815000000000001,"5min":20.047999999999998,"15min":24.436},"min":{"1min":8.5180000000000007,"5min":7.6029999999999998,"15min":7.1189999999999998},"max":{"1min":83.149000000000001,"5min":155.02000000000001,"15min":236.83500000000001},"last":13.298999999999999}]},{"osd":6,"last update":"Mon Oct 30 17:55:23 2023","interfaces":[{"interface":"back","average":{"1min":19.027000000000001,"5min":20.858000000000001,"15min":24.974},"min":{"1min":8.2159999999999993,"5min":8.2159999999999993,"15min":8.2159999999999993},"max":{"1min":80.769999999999996,"5min":125.158,"15min":246.01499999999999},"last":8.2159999999999993},{"interface":"front","average":{"1min":18.800999999999998,"5min":22.594999999999999,"15min":26.859999999999999},"min":{"1min":10.052,"5min":6.8449999999999998,"15min":6.8449999999999998},"max":{"1min":79.447000000000003,"5min":189.01599999999999,"15min":191.96700000000001},"last":16.526}]},{"osd":7,"last update":"Mon Oct 30 17:55:20 2023","interfaces":[{"interface":"back","average":{"1min":19.669,"5min":22.777999999999999,"15min":27.725000000000001},"min":{"1min":11.65,"5min":10.622999999999999,"15min":9.157},"max":{"1min":81.537000000000006,"5min":135.47800000000001,"15min":262.74200000000002},"last":14.842000000000001},{"interface":"front","average":{"1min":22.597999999999999,"5min":27.381,"15min":31.349},"min":{"1min":12.738,"5min":12.188000000000001,"15min":10.122999999999999},"max":{"1min":84.094999999999999,"5min":140.83500000000001,"15min":264.81},"last":17.352}]}]},{"osd":3,"up_from":16,"seq":68719477004,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83120,"kb_used_data":7528,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288720,"statfs":{"total":96636764160,"available":96551649280,"internally_reserved":0,"allocated":7708672,"data_stored":4236976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":30.98,"5min":20.527999999999999,"15min":16.279},"min":{"1min":5.5099999999999998,"5min":4.4489999999999998,"15min":4.3869999999999996},"max":{"1min":184.12899999999999,"5min":203.85499999999999,"15min":203.85499999999999},"last":11.635},{"interface":"front","average":{"1min":33.973999999999997,"5min":21.707999999999998,"15min":18.739999999999998},"min":{"1min":6.4829999999999997,"5min":4.8449999999999998,"15min":4.367},"max":{"1min":138.00399999999999,"5min":176.83500000000001,"15min":364.05399999999997},"last":8.0109999999999992}]},{"osd":1,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":20.564,"5min":23.222000000000001,"15min":23.289000000000001},"min":{"1min":7.4489999999999998,"5min":7.1909999999999998,"15min":5.5380000000000003},"max":{"1min":124.999,"5min":178.04300000000001,"15min":190.55199999999999},"last":9.8109999999999999},{"interface":"front","average":{"1min":19.936,"5min":23.527000000000001,"15min":22.817},"min":{"1min":7.7690000000000001,"5min":5.9290000000000003,"15min":5.609},"max":{"1min":105.627,"5min":210.381,"15min":240.64500000000001},"last":6.1230000000000002}]},{"osd":2,"last update":"Mon Oct 30 17:55:21 2023","interfaces":[{"interface":"back","average":{"1min":24.437000000000001,"5min":29.425000000000001,"15min":23.103999999999999},"min":{"1min":8.4120000000000008,"5min":7.2969999999999997,"15min":5.234},"max":{"1min":116.946,"5min":251.941,"15min":397.50099999999998},"last":12.361000000000001},{"interface":"front","average":{"1min":14.188000000000001,"5min":21.253,"15min":23},"min":{"1min":6.5750000000000002,"5min":6.54,"15min":6.54},"max":{"1min":68.379999999999995,"5min":147.50399999999999,"15min":219.65000000000001},"last":11.047000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":24.489999999999998,"5min":34.338000000000001,"15min":26.545000000000002},"min":{"1min":9.0999999999999996,"5min":7.7729999999999997,"15min":5.8140000000000001},"max":{"1min":126.723,"5min":318.92200000000003,"15min":318.92200000000003},"last":8.9900000000000002},{"interface":"front","average":{"1min":26.800000000000001,"5min":27.832999999999998,"15min":21.242999999999999},"min":{"1min":8.5909999999999993,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":137.06999999999999,"5min":268.01600000000002,"15min":268.01600000000002},"last":13.925000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":43.463000000000001,"5min":26.356000000000002,"15min":23.449000000000002},"min":{"1min":10.387,"5min":7.7569999999999997,"15min":7.7569999999999997},"max":{"1min":217.596,"5min":217.596,"15min":221.821},"last":10.449999999999999},{"interface":"front","average":{"1min":35.868000000000002,"5min":30.478999999999999,"15min":26.512},"min":{"1min":9.6460000000000008,"5min":7.2009999999999996,"15min":7.2009999999999996},"max":{"1min":223.203,"5min":225.15299999999999,"15min":225.15299999999999},"last":16.361999999999998}]},{"osd":6,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":33.683999999999997,"5min":31.859000000000002,"15min":26.734000000000002},"min":{"1min":11.108000000000001,"5min":7.9589999999999996,"15min":7.7439999999999998},"max":{"1min":137.48599999999999,"5min":272.536,"15min":272.536},"last":17.393000000000001},{"interface":"front","average":{"1min":30.719000000000001,"5min":24.899999999999999,"15min":23.899000000000001},"min":{"1min":9.8620000000000001,"5min":9.3040000000000003,"15min":7.524},"max":{"1min":121.95999999999999,"5min":162.61799999999999,"15min":199.892},"last":18.219999999999999}]},{"osd":7,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":28.047000000000001,"5min":30.155000000000001,"15min":28.039000000000001},"min":{"1min":12.372999999999999,"5min":9.8399999999999999,"15min":9.7059999999999995},"max":{"1min":124.315,"5min":196.05699999999999,"15min":196.66399999999999},"last":15.763999999999999},{"interface":"front","average":{"1min":25.338000000000001,"5min":29.327999999999999,"15min":28.225000000000001},"min":{"1min":8.7590000000000003,"5min":8.7590000000000003,"15min":8.7590000000000003},"max":{"1min":157.10499999999999,"5min":182.07499999999999,"15min":221.59100000000001},"last":14.878}]}]},{"osd":1,"up_from":16,"seq":68719477015,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61300,"kb_used_data":7916,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310540,"statfs":{"total":96636764160,"available":96573992960,"internally_reserved":0,"allocated":8105984,"data_stored":4634811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:17 2023","interfaces":[{"interface":"back","average":{"1min":9.6950000000000003,"5min":12.606,"15min":16.495999999999999},"min":{"1min":5.2350000000000003,"5min":5.0149999999999997,"15min":3.8279999999999998},"max":{"1min":16.949000000000002,"5min":162.364,"15min":313.69400000000002},"last":13.801},{"interface":"front","average":{"1min":12.805,"5min":13.266,"15min":16.631},"min":{"1min":7.0709999999999997,"5min":5.9829999999999997,"15min":4.9089999999999998},"max":{"1min":37.804000000000002,"5min":59.277000000000001,"15min":209.62799999999999},"last":9.4849999999999994}]},{"osd":2,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":15.744999999999999,"15min":19.428999999999998},"min":{"1min":9.1430000000000007,"5min":6.6139999999999999,"15min":6.0279999999999996},"max":{"1min":48.945,"5min":84.930999999999997,"15min":267.83499999999998},"last":8.3399999999999999},{"interface":"front","average":{"1min":18.347000000000001,"5min":18.187000000000001,"15min":18.768000000000001},"min":{"1min":10.164,"5min":7.0270000000000001,"15min":5.8550000000000004},"max":{"1min":60.828000000000003,"5min":89.875,"15min":178.44},"last":10.209}]},{"osd":3,"last update":"Mon Oct 30 17:55:20 2023","interfaces":[{"interface":"back","average":{"1min":12.201000000000001,"5min":246.72800000000001,"15min":93.194000000000003},"min":{"1min":5.5549999999999997,"5min":5.5549999999999997,"15min":5.2990000000000004},"max":{"1min":68.070999999999998,"5min":8788.2559999999994,"15min":8788.2559999999994},"last":8.6989999999999998},{"interface":"front","average":{"1min":17.672999999999998,"5min":247.571,"15min":96.239999999999995},"min":{"1min":10.819000000000001,"5min":10.159000000000001,"15min":6.8819999999999997},"max":{"1min":41.433999999999997,"5min":8677.7369999999992,"15min":8677.7369999999992},"last":10.855}]},{"osd":4,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":15.423999999999999,"5min":16.965,"15min":21.763999999999999},"min":{"1min":7.0250000000000004,"5min":6.8220000000000001,"15min":6.8220000000000001},"max":{"1min":49.585999999999999,"5min":91.501999999999995,"15min":235.125},"last":15.952999999999999},{"interface":"front","average":{"1min":26.652999999999999,"5min":22.824000000000002,"15min":23.303999999999998},"min":{"1min":11.677,"5min":10.278,"15min":7.673},"max":{"1min":118.545,"5min":118.545,"15min":334.19900000000001},"last":17.59}]},{"osd":5,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":25.949999999999999,"5min":19.481999999999999,"15min":25.600000000000001},"min":{"1min":10.999000000000001,"5min":9.0389999999999997,"15min":7.6360000000000001},"max":{"1min":93.442999999999998,"5min":93.442999999999998,"15min":181.976},"last":16.808},{"interface":"front","average":{"1min":24.221,"5min":21.082999999999998,"15min":28.675999999999998},"min":{"1min":12.029,"5min":9.3759999999999994,"15min":8.0869999999999997},"max":{"1min":122.42700000000001,"5min":122.42700000000001,"15min":262.61799999999999},"last":12.648}]},{"osd":6,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":24.259,"5min":17.661999999999999,"15min":22.719999999999999},"min":{"1min":6.2750000000000004,"5min":6.2750000000000004,"15min":6.2750000000000004},"max":{"1min":173.261,"5min":200.369,"15min":249.25299999999999},"last":13.398},{"interface":"front","average":{"1min":25.248999999999999,"5min":21.978999999999999,"15min":19.956},"min":{"1min":13.238,"5min":9.452,"15min":6.2880000000000003},"max":{"1min":112.628,"5min":201.756,"15min":240.72200000000001},"last":11.496}]},{"osd":7,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":25.876999999999999,"5min":22.286999999999999,"15min":25.803999999999998},"min":{"1min":10.916,"5min":9.8200000000000003,"15min":9.4269999999999996},"max":{"1min":123.173,"5min":123.173,"15min":145.078},"last":18.358000000000001},{"interface":"front","average":{"1min":21.436,"5min":23.945,"15min":28.402000000000001},"min":{"1min":14.316000000000001,"5min":11.098000000000001,"15min":10.628},"max":{"1min":39.271999999999998,"5min":203.929,"15min":203.929},"last":12.051}]}]},{"osd":2,"up_from":15,"seq":64424509718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60908,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310932,"statfs":{"total":96636764160,"available":96574394368,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.643000000000001,"5min":13.712,"15min":16.542000000000002},"min":{"1min":5.9649999999999999,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":87.680999999999997,"5min":106.839,"15min":292.57299999999998},"last":9.9499999999999993},{"interface":"front","average":{"1min":16.539999999999999,"5min":17.335999999999999,"15min":20.198},"min":{"1min":7.569,"5min":7.1319999999999997,"15min":6.7489999999999997},"max":{"1min":88.846000000000004,"5min":111.07599999999999,"15min":283.46899999999999},"last":16.395}]},{"osd":1,"last update":"Mon Oct 30 17:55:16 2023","interfaces":[{"interface":"back","average":{"1min":15.481999999999999,"5min":20.122,"15min":22.815999999999999},"min":{"1min":10.173,"5min":6.1539999999999999,"15min":6.1539999999999999},"max":{"1min":31.516999999999999,"5min":152.304,"15min":273.93599999999998},"last":15.576000000000001},{"interface":"front","average":{"1min":10.638,"5min":13.188000000000001,"15min":14.933},"min":{"1min":5.2080000000000002,"5min":4.9509999999999996,"15min":4.1890000000000001},"max":{"1min":24.187000000000001,"5min":109.767,"15min":152.87299999999999},"last":8.4339999999999993}]},{"osd":3,"last update":"Mon Oct 30 17:55:16 2023","interfaces":[{"interface":"back","average":{"1min":15.285,"5min":220.51400000000001,"15min":84.436000000000007},"min":{"1min":7.0019999999999998,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":108.666,"5min":8749.8639999999996,"15min":8749.8639999999996},"last":7.8250000000000002},{"interface":"front","average":{"1min":21.670000000000002,"5min":226.71199999999999,"15min":90.552999999999997},"min":{"1min":11.273999999999999,"5min":7.7859999999999996,"15min":6.6799999999999997},"max":{"1min":109.779,"5min":8752.7450000000008,"15min":8752.7450000000008},"last":14.468}]},{"osd":4,"last update":"Mon Oct 30 17:55:11 2023","interfaces":[{"interface":"back","average":{"1min":14.714,"5min":20.983000000000001,"15min":25.411000000000001},"min":{"1min":8.6059999999999999,"5min":8.5990000000000002,"15min":8.5990000000000002},"max":{"1min":36.32,"5min":124.502,"15min":301.34399999999999},"last":17.969000000000001},{"interface":"front","average":{"1min":18.843,"5min":26.375,"15min":30.210999999999999},"min":{"1min":10.959,"5min":10.959,"15min":10.959},"max":{"1min":36.787999999999997,"5min":134.44800000000001,"15min":300.68900000000002},"last":17.204999999999998}]},{"osd":5,"last update":"Mon Oct 30 17:55:16 2023","interfaces":[{"interface":"back","average":{"1min":17.263000000000002,"5min":18.863,"15min":27.036000000000001},"min":{"1min":6.516,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":58.253999999999998,"5min":198.46700000000001,"15min":318.01900000000001},"last":10.551},{"interface":"front","average":{"1min":24.870000000000001,"5min":20.832999999999998,"15min":21.594999999999999},"min":{"1min":10.16,"5min":8.4760000000000009,"15min":5.8920000000000003},"max":{"1min":107.361,"5min":122.861,"15min":325.87099999999998},"last":13.82}]},{"osd":6,"last update":"Mon Oct 30 17:55:11 2023","interfaces":[{"interface":"back","average":{"1min":16.164999999999999,"5min":18.206,"15min":18.603000000000002},"min":{"1min":8.9909999999999997,"5min":8.7420000000000009,"15min":6.3979999999999997},"max":{"1min":37.514000000000003,"5min":70.927999999999997,"15min":176.274},"last":11.56},{"interface":"front","average":{"1min":19.661000000000001,"5min":24.739000000000001,"15min":28.327999999999999},"min":{"1min":9.016,"5min":9.016,"15min":9.016},"max":{"1min":39.222000000000001,"5min":133.02199999999999,"15min":306.81799999999998},"last":18.763999999999999}]},{"osd":7,"last update":"Mon Oct 30 17:55:11 2023","interfaces":[{"interface":"back","average":{"1min":12.196,"5min":15.977,"15min":20.768000000000001},"min":{"1min":8.3610000000000007,"5min":7.2549999999999999,"15min":6.9669999999999996},"max":{"1min":16.366,"5min":127.804,"15min":307.69499999999999},"last":11.263},{"interface":"front","average":{"1min":17.422000000000001,"5min":21.34,"15min":26.643000000000001},"min":{"1min":10.691000000000001,"5min":8.9550000000000001,"15min":8.5700000000000003},"max":{"1min":28.623000000000001,"5min":129.26599999999999,"15min":317.13400000000001},"last":12.156000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542421,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83120,"kb_used_data":7528,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288720,"statfs":{"total":96636764160,"available":96551649280,"internally_reserved":0,"allocated":7708672,"data_stored":4236976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77396427},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":17.451000000000001,"5min":16.484999999999999,"15min":16.552},"min":{"1min":5.1619999999999999,"5min":4.6230000000000002,"15min":4.1580000000000004},"max":{"1min":154.50899999999999,"5min":172.93199999999999,"15min":256.154},"last":5.5579999999999998},{"interface":"front","average":{"1min":16.391999999999999,"5min":17.413,"15min":16.408999999999999},"min":{"1min":5.2610000000000001,"5min":4.5999999999999996,"15min":4.3959999999999999},"max":{"1min":144.61000000000001,"5min":176.59399999999999,"15min":258.25799999999998},"last":8.5150000000000006}]},{"osd":1,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":12.489000000000001,"5min":14.763999999999999,"15min":16.366},"min":{"1min":6.2990000000000004,"5min":4.9640000000000004,"15min":4.2939999999999996},"max":{"1min":26.224,"5min":83.355000000000004,"15min":175.297},"last":9.1709999999999994},{"interface":"front","average":{"1min":22.309000000000001,"5min":18.849,"15min":21.831},"min":{"1min":8.2100000000000009,"5min":8.0640000000000001,"15min":5.0439999999999996},"max":{"1min":75.745000000000005,"5min":139.392,"15min":334.67599999999999},"last":6.5750000000000002}]},{"osd":2,"last update":"Mon Oct 30 17:55:05 2023","interfaces":[{"interface":"back","average":{"1min":20.262,"5min":18.870999999999999,"15min":20.088000000000001},"min":{"1min":7.7910000000000004,"5min":6.3410000000000002,"15min":6.3140000000000001},"max":{"1min":145.50800000000001,"5min":145.50800000000001,"15min":403.90899999999999},"last":12.996},{"interface":"front","average":{"1min":26.34,"5min":26.309999999999999,"15min":25.690999999999999},"min":{"1min":9.641,"5min":8.4559999999999995,"15min":7.6829999999999998},"max":{"1min":146.65100000000001,"5min":276.67700000000002,"15min":354.34800000000001},"last":14.266}]},{"osd":3,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":24.350000000000001,"5min":166.03899999999999,"15min":69.540999999999997},"min":{"1min":8.3469999999999995,"5min":8.3469999999999995,"15min":6.8849999999999998},"max":{"1min":108.26600000000001,"5min":8740.9120000000003,"15min":8740.9120000000003},"last":17.734000000000002},{"interface":"front","average":{"1min":22.797000000000001,"5min":163.64400000000001,"15min":67.997},"min":{"1min":7.6699999999999999,"5min":7.3840000000000003,"15min":6.1120000000000001},"max":{"1min":109.111,"5min":8827.2800000000007,"15min":8827.2800000000007},"last":15.089}]},{"osd":4,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":25.326000000000001,"5min":21.077000000000002,"15min":23.256},"min":{"1min":7.4109999999999996,"5min":6.758,"15min":6.758},"max":{"1min":107.45399999999999,"5min":108.28100000000001,"15min":262.92899999999997},"last":10.465},{"interface":"front","average":{"1min":28.277000000000001,"5min":27.474,"15min":25.882000000000001},"min":{"1min":9.4670000000000005,"5min":9.4670000000000005,"15min":8.5839999999999996},"max":{"1min":112.31100000000001,"5min":204.27799999999999,"15min":297.99900000000002},"last":18.52}]},{"osd":5,"last update":"Mon Oct 30 17:55:05 2023","interfaces":[{"interface":"back","average":{"1min":28.478000000000002,"5min":24.763999999999999,"15min":27.021999999999998},"min":{"1min":11.518000000000001,"5min":9.4339999999999993,"15min":7.7729999999999997},"max":{"1min":173.75999999999999,"5min":235.08500000000001,"15min":298.197},"last":11.068},{"interface":"front","average":{"1min":19.510000000000002,"5min":21.577000000000002,"15min":21.898},"min":{"1min":10.781000000000001,"5min":6.5010000000000003,"15min":5.5010000000000003},"max":{"1min":79.013000000000005,"5min":266.30200000000002,"15min":280.34100000000001},"last":15.954000000000001}]},{"osd":6,"last update":"Mon Oct 30 17:55:05 2023","interfaces":[{"interface":"back","average":{"1min":26.076000000000001,"5min":23.91,"15min":23.134},"min":{"1min":8.6750000000000007,"5min":8.6750000000000007,"15min":6.9400000000000004},"max":{"1min":122.34099999999999,"5min":174.517,"15min":260.10399999999998},"last":16.872},{"interface":"front","average":{"1min":25.681999999999999,"5min":26.704999999999998,"15min":26.986999999999998},"min":{"1min":8.7430000000000003,"5min":8.7430000000000003,"15min":8.0060000000000002},"max":{"1min":79.855000000000004,"5min":140.06399999999999,"15min":257.62599999999998},"last":11.632999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542424,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78956,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292884,"statfs":{"total":96636764160,"available":96555913216,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":12.414999999999999,"5min":15.685,"15min":16.145},"min":{"1min":4.617,"5min":4.3339999999999996,"15min":4.1840000000000002},"max":{"1min":100.479,"5min":135.98099999999999,"15min":254.899},"last":15.516999999999999},{"interface":"front","average":{"1min":16.834,"5min":16.405999999999999,"15min":18.106999999999999},"min":{"1min":5.2670000000000003,"5min":5.2670000000000003,"15min":5.2670000000000003},"max":{"1min":159.16,"5min":159.16,"15min":270.51600000000002},"last":10.602}]},{"osd":1,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":18.103999999999999,"5min":19.869,"15min":20.129999999999999},"min":{"1min":7.3620000000000001,"5min":7.2350000000000003,"15min":5.1120000000000001},"max":{"1min":122.685,"5min":149.53899999999999,"15min":177.15799999999999},"last":12.512},{"interface":"front","average":{"1min":16.960999999999999,"5min":18.959,"15min":22.780999999999999},"min":{"1min":6.819,"5min":6.2199999999999998,"15min":5.883},"max":{"1min":119.675,"5min":221.483,"15min":309.04199999999997},"last":17.146999999999998}]},{"osd":2,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":19.047000000000001,"5min":18.303999999999998,"15min":19.811},"min":{"1min":8.1219999999999999,"5min":6.8929999999999998,"15min":5.859},"max":{"1min":145.20400000000001,"5min":145.20400000000001,"15min":231.15600000000001},"last":16.346},{"interface":"front","average":{"1min":22.48,"5min":21.637,"15min":22.283999999999999},"min":{"1min":13.409000000000001,"5min":11.173999999999999,"15min":7.1890000000000001},"max":{"1min":146.94,"5min":146.94,"15min":197.58699999999999},"last":18.001999999999999}]},{"osd":3,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":17.527000000000001,"5min":249.054,"15min":96.846999999999994},"min":{"1min":5.9400000000000004,"5min":5.5039999999999996,"15min":5.3849999999999998},"max":{"1min":152.352,"5min":7958.2420000000002,"15min":7958.2420000000002},"last":6.7590000000000003},{"interface":"front","average":{"1min":24.044,"5min":252.749,"15min":98.444000000000003},"min":{"1min":8.3620000000000001,"5min":6.7009999999999996,"15min":5.9859999999999998},"max":{"1min":142.91399999999999,"5min":7971.3900000000003,"15min":7971.3900000000003},"last":8.5939999999999994}]},{"osd":4,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":19.713999999999999,"5min":23.420999999999999,"15min":23.501999999999999},"min":{"1min":9.6539999999999999,"5min":7.5300000000000002,"15min":7.5300000000000002},"max":{"1min":80.417000000000002,"5min":236.994,"15min":236.994},"last":18.847000000000001},{"interface":"front","average":{"1min":21.596,"5min":21.725999999999999,"15min":22.199999999999999},"min":{"1min":9.5459999999999994,"5min":8.6479999999999997,"15min":6.7859999999999996},"max":{"1min":154.33000000000001,"5min":228.88200000000001,"15min":228.88200000000001},"last":11.647}]},{"osd":5,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":26.045999999999999,"5min":21.262,"15min":24.367999999999999},"min":{"1min":6.8289999999999997,"5min":6.5540000000000003,"15min":5.7809999999999997},"max":{"1min":274.59800000000001,"5min":274.59800000000001,"15min":289.22699999999998},"last":9.5869999999999997},{"interface":"front","average":{"1min":26.004000000000001,"5min":24.387,"15min":26.591000000000001},"min":{"1min":12.798,"5min":11.048999999999999,"15min":10.132999999999999},"max":{"1min":148.43799999999999,"5min":173.684,"15min":230.613},"last":19.687999999999999}]},{"osd":7,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":19.562000000000001,"5min":22.870999999999999,"15min":24.946999999999999},"min":{"1min":7.1609999999999996,"5min":7.1609999999999996,"15min":7.1609999999999996},"max":{"1min":150.018,"5min":192.72399999999999,"15min":290.22300000000001},"last":14.785},{"interface":"front","average":{"1min":23.748000000000001,"5min":25.805,"15min":27.835000000000001},"min":{"1min":10.547000000000001,"5min":9.5440000000000005,"15min":7.4489999999999998},"max":{"1min":158.352,"5min":270.01900000000001,"15min":270.01900000000001},"last":13.085000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542424,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60908,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310932,"statfs":{"total":96636764160,"available":96574394368,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":17.713999999999999,"5min":18.492999999999999,"15min":16.776},"min":{"1min":4.6639999999999997,"5min":3.7290000000000001,"15min":3.7290000000000001},"max":{"1min":84.162999999999997,"5min":278.70299999999997,"15min":278.70299999999997},"last":4.9329999999999998},{"interface":"front","average":{"1min":15.43,"5min":17.457000000000001,"15min":17.106000000000002},"min":{"1min":5.4299999999999997,"5min":4.9850000000000003,"15min":4.9850000000000003},"max":{"1min":47.259,"5min":174.08099999999999,"15min":190.619},"last":8.3510000000000009}]},{"osd":1,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":20.577999999999999,"5min":18.545000000000002,"15min":20.265999999999998},"min":{"1min":9.3889999999999993,"5min":5.6029999999999998,"15min":5.6029999999999998},"max":{"1min":58.914000000000001,"5min":136.81200000000001,"15min":184.21700000000001},"last":13.327999999999999},{"interface":"front","average":{"1min":17.042999999999999,"5min":17.469000000000001,"15min":20.419},"min":{"1min":6.3929999999999998,"5min":6.1669999999999998,"15min":6.0819999999999999},"max":{"1min":53.759,"5min":139.191,"15min":199.434},"last":9.3620000000000001}]},{"osd":2,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":22.640999999999998,"5min":20.824000000000002,"15min":22.341000000000001},"min":{"1min":9.6809999999999992,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":112.958,"5min":279.37099999999998,"15min":279.37099999999998},"last":11.321999999999999},{"interface":"front","average":{"1min":26.033000000000001,"5min":22.379000000000001,"15min":20.827999999999999},"min":{"1min":11.018000000000001,"5min":6.9390000000000001,"15min":5.8259999999999996},"max":{"1min":92.144999999999996,"5min":198.02500000000001,"15min":321.54000000000002},"last":16.946000000000002}]},{"osd":3,"last update":"Mon Oct 30 17:55:16 2023","interfaces":[{"interface":"back","average":{"1min":17.798999999999999,"5min":164.95699999999999,"15min":69.094999999999999},"min":{"1min":7.774,"5min":5.9219999999999997,"15min":5.9219999999999997},"max":{"1min":56.622999999999998,"5min":8219.1820000000007,"15min":8219.1820000000007},"last":11.93},{"interface":"front","average":{"1min":15.321,"5min":158.661,"15min":68.135000000000005},"min":{"1min":7.3440000000000003,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":42.534999999999997,"5min":8140.4719999999998,"15min":8140.4719999999998},"last":8.9830000000000005}]},{"osd":5,"last update":"Mon Oct 30 17:54:53 2023","interfaces":[{"interface":"back","average":{"1min":18.893000000000001,"5min":22.899000000000001,"15min":23.814},"min":{"1min":9.1790000000000003,"5min":6.4169999999999998,"15min":6.4169999999999998},"max":{"1min":65.632000000000005,"5min":292.22800000000001,"15min":292.22800000000001},"last":12.589},{"interface":"front","average":{"1min":18.199999999999999,"5min":22.989999999999998,"15min":22.789999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":59.558,"5min":179.67500000000001,"15min":180.28800000000001},"last":9.9670000000000005}]},{"osd":6,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":31.431000000000001,"5min":27.053999999999998,"15min":24.242999999999999},"min":{"1min":9.5459999999999994,"5min":9.5459999999999994,"15min":6.5810000000000004},"max":{"1min":126.249,"5min":191.995,"15min":191.995},"last":14.055},{"interface":"front","average":{"1min":28.806999999999999,"5min":26.370999999999999,"15min":25.456},"min":{"1min":10.333,"5min":8.2420000000000009,"15min":7.6779999999999999},"max":{"1min":86.864999999999995,"5min":207.30099999999999,"15min":207.30099999999999},"last":17.736999999999998}]},{"osd":7,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":16.986000000000001,"5min":24.992999999999999,"15min":25.995000000000001},"min":{"1min":8.6899999999999995,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":58.936999999999998,"5min":396.928,"15min":396.928},"last":10.672000000000001},{"interface":"front","average":{"1min":21.718,"5min":27.449000000000002,"15min":28.113},"min":{"1min":10.461,"5min":10.461,"15min":7.0860000000000003},"max":{"1min":55.451000000000001,"5min":366.32499999999999,"15min":366.32499999999999},"last":14.756}]}]},{"osd":5,"up_from":13,"seq":55834575128,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79348,"kb_used_data":7916,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292492,"statfs":{"total":96636764160,"available":96555511808,"internally_reserved":0,"allocated":8105984,"data_stored":4634811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":22.184999999999999,"5min":16.451000000000001,"15min":16.408999999999999},"min":{"1min":4.8639999999999999,"5min":3.9910000000000001,"15min":3.9569999999999999},"max":{"1min":154.71299999999999,"5min":154.71299999999999,"15min":155.58799999999999},"last":15.593},{"interface":"front","average":{"1min":23.321999999999999,"5min":18.834,"15min":17.321000000000002},"min":{"1min":4.6429999999999998,"5min":4.6429999999999998,"15min":4.6429999999999998},"max":{"1min":162.02000000000001,"5min":162.02000000000001,"15min":178.42500000000001},"last":9.8040000000000003}]},{"osd":1,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":30.655000000000001,"5min":18.181999999999999,"15min":20.413},"min":{"1min":5.9710000000000001,"5min":5.0970000000000004,"15min":4.8810000000000002},"max":{"1min":235.82900000000001,"5min":235.82900000000001,"15min":235.82900000000001},"last":7.1130000000000004},{"interface":"front","average":{"1min":23.542999999999999,"5min":16.102,"15min":17.175000000000001},"min":{"1min":5.6929999999999996,"5min":5.6929999999999996,"15min":5.1120000000000001},"max":{"1min":107.85899999999999,"5min":128.197,"15min":278.41300000000001},"last":17.178999999999998}]},{"osd":2,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":29.405000000000001,"5min":19.856999999999999,"15min":19.73},"min":{"1min":10.742000000000001,"5min":7.101,"15min":5.8949999999999996},"max":{"1min":232.47800000000001,"5min":232.47800000000001,"15min":234.09299999999999},"last":19.649999999999999},{"interface":"front","average":{"1min":21.835999999999999,"5min":17.140999999999998,"15min":19.927},"min":{"1min":7.3819999999999997,"5min":6.5510000000000002,"15min":6.5060000000000002},"max":{"1min":109.47799999999999,"5min":109.47799999999999,"15min":277.52300000000002},"last":9.0370000000000008}]},{"osd":3,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":34.145000000000003,"5min":154.58500000000001,"15min":65.951999999999998},"min":{"1min":6.3410000000000002,"5min":6.2080000000000002,"15min":6.2080000000000002},"max":{"1min":265.51400000000001,"5min":6785.9989999999998,"15min":6785.9989999999998},"last":12.98},{"interface":"front","average":{"1min":36.165999999999997,"5min":155.98099999999999,"15min":66.064999999999998},"min":{"1min":9.6110000000000007,"5min":6.6870000000000003,"15min":6.2839999999999998},"max":{"1min":288.66199999999998,"5min":6786.7420000000002,"15min":6786.7420000000002},"last":14.981999999999999}]},{"osd":4,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":29.448,"5min":20.707000000000001,"15min":21.905999999999999},"min":{"1min":8.2300000000000004,"5min":6.9790000000000001,"15min":6.6609999999999996},"max":{"1min":122.157,"5min":143.75800000000001,"15min":244.25},"last":18.870000000000001},{"interface":"front","average":{"1min":31.305,"5min":25.204999999999998,"15min":25.847999999999999},"min":{"1min":11.073,"5min":11.073,"15min":8.609},"max":{"1min":106.134,"5min":144.934,"15min":257.089},"last":21.451000000000001}]},{"osd":6,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":25.896999999999998,"5min":18.837,"15min":21.152000000000001},"min":{"1min":6.3680000000000003,"5min":6.3680000000000003,"15min":6.3680000000000003},"max":{"1min":169.02199999999999,"5min":169.02199999999999,"15min":280.36000000000001},"last":11.099},{"interface":"front","average":{"1min":27.088999999999999,"5min":22.541,"15min":25.475000000000001},"min":{"1min":9.0850000000000009,"5min":7.4020000000000001,"15min":7.4020000000000001},"max":{"1min":113.434,"5min":113.434,"15min":346.47000000000003},"last":13.643000000000001}]},{"osd":7,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":35.899999999999999,"5min":23.495999999999999,"15min":27.568999999999999},"min":{"1min":7.7530000000000001,"5min":7.7530000000000001,"15min":7.6520000000000001},"max":{"1min":170.19399999999999,"5min":170.19399999999999,"15min":264.64499999999998},"last":14.366},{"interface":"front","average":{"1min":28.663,"5min":23.901,"15min":27.988},"min":{"1min":10.331,"5min":10.331,"15min":9.9979999999999993},"max":{"1min":96.566999999999993,"5min":145.00899999999999,"15min":282.32400000000001},"last":20.628}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T17:55:31.966 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:31.966 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:31.967 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:31.967 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:31.967 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:31.968 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2023-10-30T17:33:34.187727+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=33, tm_sec=34, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=17, tm_min=55, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T17:55:31.968 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T17:55:39.541 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:23.507 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:39.544 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:23.510 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:39.557 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:23.523 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:39.586 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:23.552 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:39.587 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:23.526 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:39.590 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:23.529 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:39.591 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:23.557 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:39.602 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:23.541 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:39.606 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:23.573 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:49.582 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:33.547 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:49.584 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:33.550 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:49.598 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:33.564 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:49.630 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:33.569 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:49.632 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:33.598 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:49.633 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:33.573 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:49.635 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:33.601 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:49.646 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:33.585 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:49.647 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:33.613 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:51.969 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T17:55:52.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.067+0000 7f5ee6e07640 10 monclient: get_monmap_and_config 2023-10-30T17:55:52.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.067+0000 7f5ee6e07640 10 monclient: build_initial_monmap 2023-10-30T17:55:52.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.067+0000 7f5ee6e07640 10 monclient: monmap: 2023-10-30T17:55:52.069 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:52.069 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:52.069 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:52.068350+0000 2023-10-30T17:55:52.070 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:52.068350+0000 2023-10-30T17:55:52.070 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:52.070 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.070 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:52.070 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:52.071 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:52.071 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:52.071 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:52.071 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 1 Processor -- start 2023-10-30T17:55:52.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 1 -- start start 2023-10-30T17:55:52.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 10 monclient: init 2023-10-30T17:55:52.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:52.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-10-30T17:55:52.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5ee006bbf0 0x7f5ee006bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee006bbf0 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:52.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ee0071b60 addr v1:172.21.15.177:6789/0 2023-10-30T17:55:52.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5ee00708a0 0x7f5ee0070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee00708a0 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:52.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee006cf10 con 0x7f5ee0071b60 2023-10-30T17:55:52.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee006fec0 con 0x7f5ee00708a0 2023-10-30T17:55:52.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee00701a0 con 0x7f5ee006bbf0 2023-10-30T17:55:52.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:52.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.068+0000 7f5ee6e07640 10 monclient(hunting): authenticate will time out at 2734.391785s 2023-10-30T17:55:52.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.069+0000 7f5ee53a4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5ee00708a0 0x7f5ee0070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.069+0000 7f5ee4ba3640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5ee006bbf0 0x7f5ee006bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.069+0000 7f5ee53a4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5ee00708a0 0x7f5ee0070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:45530/0 (socket says 172.21.15.170:45530) 2023-10-30T17:55:52.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.069+0000 7f5ee53a4640 1 -- 172.21.15.170:0/2349330728 learned_addr learned my addr 172.21.15.170:0/2349330728 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:52.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.069+0000 7f5ee53a4640 10 monclient(hunting): get_auth_request con 0x7f5ee00708a0 auth_method 0 2023-10-30T17:55:52.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.069+0000 7f5ee53a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.069+0000 7f5ee53a4640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.069+0000 7f5ee53a4640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.070+0000 7f5ee53a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.070+0000 7f5ee53a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.070+0000 7f5ee53a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.070+0000 7f5ee4ba3640 10 monclient(hunting): get_auth_request con 0x7f5ee006bbf0 auth_method 0 2023-10-30T17:55:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.070+0000 7f5ee4ba3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.070+0000 7f5ee4ba3640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.070+0000 7f5ee4ba3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.070+0000 7f5ee4ba3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.070+0000 7f5ee4ba3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.070+0000 7f5ee4ba3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.071+0000 7f5ee53a4640 10 monclient(hunting): handle_auth_done global_id 16092 payload 274 2023-10-30T17:55:52.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.071+0000 7f5ee53a4640 1 -- 172.21.15.170:0/2349330728 >> v2:172.21.15.177:3300/0 conn(0x7f5ee006bbf0 msgr2=0x7f5ee006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:52.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.071+0000 7f5ee53a4640 1 --2- 172.21.15.170:0/2349330728 >> v2:172.21.15.177:3300/0 conn(0x7f5ee006bbf0 0x7f5ee006bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.071+0000 7f5ee53a4640 1 -- 172.21.15.170:0/2349330728 >> v1:172.21.15.177:6789/0 conn(0x7f5ee0071b60 legacy=0x7f5ee00afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:52.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.071+0000 7f5ee53a4640 10 monclient: _finish_hunting 0 2023-10-30T17:55:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.071+0000 7f5ee53a4640 1 monclient: found mon.noname-a 2023-10-30T17:55:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.071+0000 7f5ee53a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.071+0000 7f5ee53a4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.071+0000 7f5ee53a4640 1 -- 172.21.15.170:0/2349330728 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee006c990 con 0x7f5ee00708a0 2023-10-30T17:55:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.073+0000 7f5ee53a4640 1 --2- 172.21.15.170:0/2349330728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 0x7f5ee0070d20 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7f5ee006b6d0 tx=0x7f5ed802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=776f19c3604e3fc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:52.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.076+0000 7f5edf7fe640 1 -- 172.21.15.170:0/2349330728 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed802c9f0 con 0x7f5ee00708a0 2023-10-30T17:55:52.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.076+0000 7f5edf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:52.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.076+0000 7f5edf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:52.088 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.076+0000 7f5edf7fe640 10 monclient: dump: 2023-10-30T17:55:52.088 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:52.088 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:52.089 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.089 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.089 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:52.089 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.089 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:52.089 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:52.090 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:52.090 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.076+0000 7f5edf7fe640 10 monclient: _finish_auth 0 2023-10-30T17:55:52.090 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.076+0000 7f5edf7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:55:52.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.076+0000 7f5edf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:52.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.076+0000 7f5ee6e07640 5 monclient: authenticate success, global_id 16092 2023-10-30T17:55:52.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.076+0000 7f5ee6e07640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:52.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.077+0000 7f5edf7fe640 1 -- 172.21.15.170:0/2349330728 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed802cb50 con 0x7f5ee00708a0 2023-10-30T17:55:52.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.077+0000 7f5edf7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:52.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.077+0000 7f5edf7fe640 1 -- 172.21.15.170:0/2349330728 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed80343f0 con 0x7f5ee00708a0 2023-10-30T17:55:52.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.077+0000 7f5edf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:52.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.077+0000 7f5edf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:52.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.077+0000 7f5edf7fe640 10 monclient: dump: 2023-10-30T17:55:52.095 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:52.095 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:52.095 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.095 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.095 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:52.095 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.096 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:52.096 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:52.096 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:52.096 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.077+0000 7f5ee6e07640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:52.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.077+0000 7f5ee6e07640 4 set_mon_vals no callback set 2023-10-30T17:55:52.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.077+0000 7f5ee6e07640 10 monclient: shutdown 2023-10-30T17:55:52.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.078+0000 7f5ee6e07640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:52.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.078+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2349330728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 msgr2=0x7f5ee0070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:52.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.078+0000 7f5ee6e07640 1 --2- 172.21.15.170:0/2349330728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 0x7f5ee0070d20 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7f5ee006b6d0 tx=0x7f5ed802be50 comp rx=0 tx=0).stop 2023-10-30T17:55:52.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.078+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2349330728 shutdown_connections 2023-10-30T17:55:52.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.078+0000 7f5ee6e07640 1 --2- 172.21.15.170:0/2349330728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 0x7f5ee0070d20 unknown :-1 s=CLOSED pgs=3387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.078+0000 7f5ee6e07640 1 --2- 172.21.15.170:0/2349330728 >> v2:172.21.15.177:3300/0 conn(0x7f5ee006bbf0 0x7f5ee006bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.078+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2349330728 >> 172.21.15.170:0/2349330728 conn(0x7f5ee00b53a0 msgr2=0x7f5ee00b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:52.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.078+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2349330728 shutdown_connections 2023-10-30T17:55:52.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.078+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2349330728 wait complete. 2023-10-30T17:55:52.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.079+0000 7f5ee6e07640 10 monclient: build_initial_monmap 2023-10-30T17:55:52.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.079+0000 7f5ee6e07640 10 monclient: monmap: 2023-10-30T17:55:52.099 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:52.099 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:52.099 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:52.100 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:52.100 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:52.100 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.100 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:52.100 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:52.101 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:52.101 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.079+0000 7f5ee6e07640 1 librados: starting msgr at 2023-10-30T17:55:52.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.079+0000 7f5ee6e07640 1 librados: starting objecter 2023-10-30T17:55:52.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.079+0000 7f5ee6e07640 1 Processor -- start 2023-10-30T17:55:52.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.079+0000 7f5ee6e07640 1 -- start start 2023-10-30T17:55:52.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.079+0000 7f5ee6e07640 1 librados: setting wanted keys 2023-10-30T17:55:52.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.079+0000 7f5ee6e07640 1 librados: calling monclient init 2023-10-30T17:55:52.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.079+0000 7f5ee6e07640 10 monclient: init 2023-10-30T17:55:52.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:52.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T17:55:52.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ee006bbf0 0x7f5ee01c05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee006bbf0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:52.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 0x7f5ee01c4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee00708a0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:52.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ee0071b60 0x7f5ee01c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee0071b60 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:52.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee01c8de0 con 0x7f5ee00708a0 2023-10-30T17:55:52.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee01da020 con 0x7f5ee006bbf0 2023-10-30T17:55:52.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee01da1e0 con 0x7f5ee0071b60 2023-10-30T17:55:52.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:52.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee6e07640 10 monclient(hunting): authenticate will time out at 2734.403302s 2023-10-30T17:55:52.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5edffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 0x7f5ee01c4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee4ba3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ee006bbf0 0x7f5ee01c05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee53a4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ee0071b60 0x7f5ee01c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5edffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 0x7f5ee01c4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:45546/0 (socket says 172.21.15.170:45546) 2023-10-30T17:55:52.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5edffff640 1 -- 172.21.15.170:0/2971891271 learned_addr learned my addr 172.21.15.170:0/2971891271 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:52.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5edffff640 10 monclient(hunting): get_auth_request con 0x7f5ee00708a0 auth_method 0 2023-10-30T17:55:52.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5edffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5edffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5edffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee4ba3640 10 monclient(hunting): get_auth_request con 0x7f5ee006bbf0 auth_method 0 2023-10-30T17:55:52.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee4ba3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee4ba3640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.080+0000 7f5ee4ba3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5ee53a4640 10 monclient(hunting): get_auth_request con 0x7f5ee0071b60 auth_method 0 2023-10-30T17:55:52.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5ee53a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5ee53a4640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5ee53a4640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5edffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5edffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5edffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5ee4ba3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5ee4ba3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5ee4ba3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5ee53a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5ee53a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.081+0000 7f5ee53a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.084+0000 7f5edffff640 10 monclient(hunting): handle_auth_done global_id 16095 payload 824 2023-10-30T17:55:52.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.084+0000 7f5edffff640 1 -- 172.21.15.170:0/2971891271 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ee0071b60 msgr2=0x7f5ee01c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:52.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.084+0000 7f5edffff640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ee0071b60 0x7f5ee01c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.084+0000 7f5edffff640 1 -- 172.21.15.170:0/2971891271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ee006bbf0 msgr2=0x7f5ee01c05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:52.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.084+0000 7f5edffff640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ee006bbf0 0x7f5ee01c05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.084+0000 7f5edffff640 10 monclient: _finish_hunting 0 2023-10-30T17:55:52.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.084+0000 7f5edffff640 1 monclient: found mon.noname-a 2023-10-30T17:55:52.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.084+0000 7f5edffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:52.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.084+0000 7f5edffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:52.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.084+0000 7f5edffff640 1 -- 172.21.15.170:0/2971891271 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed8009b30 con 0x7f5ee00708a0 2023-10-30T17:55:52.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.086+0000 7f5edffff640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 0x7f5ee01c4140 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00b3c0 tx=0x7f5ecc00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:52.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.086+0000 7f5edd7fa640 10 client.?.objecter ms_handle_connect 0x7f5ee00708a0 2023-10-30T17:55:52.112 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.086+0000 7f5edd7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:52.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 1 -- 172.21.15.170:0/2971891271 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc011830 con 0x7f5ee00708a0 2023-10-30T17:55:52.114 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:52.114 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:52.114 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 10 monclient: dump: 2023-10-30T17:55:52.114 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:52.115 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:52.115 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.115 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.115 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:52.115 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.116 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:52.116 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:52.116 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:52.116 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 10 monclient: _finish_auth 0 2023-10-30T17:55:52.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 10 monclient: _check_auth_tickets 2023-10-30T17:55:52.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:52.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 1 -- 172.21.15.170:0/2971891271 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc011990 con 0x7f5ee00708a0 2023-10-30T17:55:52.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:52.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 1 -- 172.21.15.170:0/2971891271 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc011d80 con 0x7f5ee00708a0 2023-10-30T17:55:52.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:52.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:52.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edd7fa640 10 monclient: dump: 2023-10-30T17:55:52.118 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:52.118 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:52.118 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.119 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.119 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:52.119 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.119 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:52.119 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:52.119 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:52.120 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5ee6e07640 5 monclient: authenticate success, global_id 16095 2023-10-30T17:55:52.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5ee6e07640 10 monclient: _renew_subs 2023-10-30T17:55:52.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5ee6e07640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:52.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2971891271 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee01da400 con 0x7f5ee00708a0 2023-10-30T17:55:52.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5edeffd640 4 set_mon_vals no callback set 2023-10-30T17:55:52.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5ee6e07640 10 client.16095.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:52.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5ee6e07640 10 monclient: _renew_subs 2023-10-30T17:55:52.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5ee6e07640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:52.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2971891271 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee01da8c0 con 0x7f5ee00708a0 2023-10-30T17:55:52.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.090+0000 7f5ee6e07640 1 librados: init done 2023-10-30T17:55:52.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.091+0000 7f5ee6e07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:52.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.091+0000 7f5ee6e07640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:52.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.091+0000 7f5ee6e07640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:52.122 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.091+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2971891271 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee01d9db0 con 0x7f5ee00708a0 2023-10-30T17:55:52.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.331+0000 7f5edd7fa640 1 -- 172.21.15.170:0/2971891271 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5ecc004020 con 0x7f5ee00708a0 2023-10-30T17:55:52.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.331+0000 7f5edd7fa640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f5ec406c880 0x7f5ec406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.332+0000 7f5ee4ba3640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f5ec406c880 0x7f5ec406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.332+0000 7f5ee4ba3640 10 monclient: get_auth_request con 0x7f5ec406c880 auth_method 0 2023-10-30T17:55:52.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.332+0000 7f5edd7fa640 10 client.16095.objecter ms_handle_connect 0x7f5ec406c880 2023-10-30T17:55:52.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.332+0000 7f5ee4ba3640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f5ec406c880 0x7f5ec406ed40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009b40 tx=0x7f5ed4009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:52.348 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.347+0000 7f5edd7fa640 1 -- 172.21.15.170:0/2971891271 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(914..914 src has 737..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f5ecc0584b0 con 0x7f5ee00708a0 2023-10-30T17:55:52.348 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.347+0000 7f5edd7fa640 10 client.16095.objecter ms_dispatch 0x7f5ee0000b90 osd_map(914..914 src has 737..914) v4 2023-10-30T17:55:52.348 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.347+0000 7f5edd7fa640 3 client.16095.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:52.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.347+0000 7f5edd7fa640 3 client.16095.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:52.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.347+0000 7f5edd7fa640 20 client.16095.objecter dump_active .. 0 homeless 2023-10-30T17:55:52.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.347+0000 7f5edd7fa640 1 -- 172.21.15.170:0/2971891271 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5ecc0547f0 con 0x7f5ee00708a0 2023-10-30T17:55:52.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.347+0000 7f5edd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:52.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.347+0000 7f5edd7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:52.512 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.511+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2971891271 --> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ee01c8f20 con 0x7f5ec406c880 2023-10-30T17:55:52.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.516+0000 7f5edd7fa640 1 -- 172.21.15.170:0/2971891271 <== mgr.4111 v2:172.21.15.177:6800/1894191478 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152303 (secure 0 0 0) 0x7f5ee01c8f20 con 0x7f5ec406c880 2023-10-30T17:55:52.519 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-30T17:55:52.520 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2023-10-30T17:55:52.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.518+0000 7f5ee6e07640 10 librados: watch_flush enter 2023-10-30T17:55:52.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.518+0000 7f5ee6e07640 10 librados: watch_flush exit 2023-10-30T17:55:52.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.518+0000 7f5ee6e07640 20 client.16095.objecter shutdown clearing up homeless session... 2023-10-30T17:55:52.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.518+0000 7f5ee6e07640 10 client.16095.objecter successfully canceled tick 2023-10-30T17:55:52.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.518+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2971891271 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f5ec406c880 msgr2=0x7f5ec406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:52.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.518+0000 7f5ee6e07640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f5ec406c880 0x7f5ec406ed40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009b40 tx=0x7f5ed4009190 comp rx=0 tx=0).stop 2023-10-30T17:55:52.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.518+0000 7f5ee6e07640 10 monclient: shutdown 2023-10-30T17:55:52.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.518+0000 7f5ee6e07640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:52.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.518+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2971891271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 msgr2=0x7f5ee01c4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:52.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.518+0000 7f5ee6e07640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 0x7f5ee01c4140 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00b3c0 tx=0x7f5ecc00b880 comp rx=0 tx=0).stop 2023-10-30T17:55:52.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.519+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2971891271 shutdown_connections 2023-10-30T17:55:52.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.519+0000 7f5ee6e07640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f5ec406c880 0x7f5ec406ed40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.519+0000 7f5ee6e07640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ee0071b60 0x7f5ee01c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.519+0000 7f5ee6e07640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ee00708a0 0x7f5ee01c4140 unknown :-1 s=CLOSED pgs=3388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.519+0000 7f5ee6e07640 1 --2- 172.21.15.170:0/2971891271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ee006bbf0 0x7f5ee01c05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.519+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2971891271 >> 172.21.15.170:0/2971891271 conn(0x7f5ee00b53a0 msgr2=0x7f5ee01c93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:52.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.519+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2971891271 shutdown_connections 2023-10-30T17:55:52.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.519+0000 7f5ee6e07640 1 -- 172.21.15.170:0/2971891271 wait complete. 2023-10-30T17:55:52.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.519+0000 7f5ee6e07640 1 librados: shutdown 2023-10-30T17:55:52.531 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2023-10-30T17:55:51.336851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":568568,"kb_used_data":60984,"kb_used_omap":12,"kb_used_meta":507507,"kb_avail":754406152,"statfs":{"total":773094113280,"available":772511899648,"internally_reserved":0,"allocated":62447616,"data_stored":34691458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":519687736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002783"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:09.591816+0000","last_change":"2023-10-30T17:55:09.591816+0000","last_active":"2023-10-30T17:55:09.591816+0000","last_peered":"2023-10-30T17:55:09.591816+0000","last_clean":"2023-10-30T17:55:09.591816+0000","last_became_active":"2023-10-30T17:33:37.643213+0000","last_became_peered":"2023-10-30T17:33:37.643213+0000","last_unstale":"2023-10-30T17:55:09.591816+0000","last_undegraded":"2023-10-30T17:55:09.591816+0000","last_fullsized":"2023-10-30T17:55:09.591816+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:09.589496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:09.589496+0000","last_clean_scrub_stamp":"2023-10-30T17:55:09.589496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:04:42.753690+0000","scrub_duration":0.046746827999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:11.164191+0000","last_change":"2023-10-30T17:55:11.164191+0000","last_active":"2023-10-30T17:55:11.164191+0000","last_peered":"2023-10-30T17:55:11.164191+0000","last_clean":"2023-10-30T17:55:11.164191+0000","last_became_active":"2023-10-30T17:33:37.827806+0000","last_became_peered":"2023-10-30T17:33:37.827806+0000","last_unstale":"2023-10-30T17:55:11.164191+0000","last_undegraded":"2023-10-30T17:55:11.164191+0000","last_fullsized":"2023-10-30T17:55:11.164191+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:11.161754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:11.161754+0000","last_clean_scrub_stamp":"2023-10-30T17:55:11.161754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:38.406882+0000","scrub_duration":0.050985619000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:23.765774+0000","last_change":"2023-10-30T17:55:23.767753+0000","last_active":"2023-10-30T17:55:23.765774+0000","last_peered":"2023-10-30T17:55:23.765774+0000","last_clean":"2023-10-30T17:55:23.765774+0000","last_became_active":"2023-10-30T17:33:40.511287+0000","last_became_peered":"2023-10-30T17:33:40.511287+0000","last_unstale":"2023-10-30T17:55:23.765774+0000","last_undegraded":"2023-10-30T17:55:23.765774+0000","last_fullsized":"2023-10-30T17:55:23.765774+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:23.763969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:23.763969+0000","last_clean_scrub_stamp":"2023-10-30T17:55:23.763969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:40:10.265375+0000","scrub_duration":0.099663560999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":863,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:15.166708+0000","last_change":"2023-10-30T17:55:15.170251+0000","last_active":"2023-10-30T17:55:15.166708+0000","last_peered":"2023-10-30T17:55:15.166708+0000","last_clean":"2023-10-30T17:55:15.166708+0000","last_became_active":"2023-10-30T17:33:38.581457+0000","last_became_peered":"2023-10-30T17:33:38.581457+0000","last_unstale":"2023-10-30T17:55:15.166708+0000","last_undegraded":"2023-10-30T17:55:15.166708+0000","last_fullsized":"2023-10-30T17:55:15.166708+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:15.165342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:15.165342+0000","last_clean_scrub_stamp":"2023-10-30T17:55:15.165342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:50.836306+0000","scrub_duration":0.048933965000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":870,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:16.497473+0000","last_change":"2023-10-30T17:55:16.498710+0000","last_active":"2023-10-30T17:55:16.497473+0000","last_peered":"2023-10-30T17:55:16.497473+0000","last_clean":"2023-10-30T17:55:16.497473+0000","last_became_active":"2023-10-30T17:33:46.629442+0000","last_became_peered":"2023-10-30T17:33:46.629442+0000","last_unstale":"2023-10-30T17:55:16.497473+0000","last_undegraded":"2023-10-30T17:55:16.497473+0000","last_fullsized":"2023-10-30T17:55:16.497473+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:16.496140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:16.496140+0000","last_clean_scrub_stamp":"2023-10-30T17:55:16.496140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:35:16.307896+0000","scrub_duration":0.044569352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:24.718152+0000","last_change":"2023-10-30T17:55:24.719229+0000","last_active":"2023-10-30T17:55:24.718152+0000","last_peered":"2023-10-30T17:55:24.718152+0000","last_clean":"2023-10-30T17:55:24.718152+0000","last_became_active":"2023-10-30T17:33:39.135750+0000","last_became_peered":"2023-10-30T17:33:39.135750+0000","last_unstale":"2023-10-30T17:55:24.718152+0000","last_undegraded":"2023-10-30T17:55:24.718152+0000","last_fullsized":"2023-10-30T17:55:24.718152+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:24.716758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:24.716758+0000","last_clean_scrub_stamp":"2023-10-30T17:55:24.716758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:48.459846+0000","scrub_duration":0.047575569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:25.744263+0000","last_change":"2023-10-30T17:55:25.748963+0000","last_active":"2023-10-30T17:55:25.744263+0000","last_peered":"2023-10-30T17:55:25.744263+0000","last_clean":"2023-10-30T17:55:25.744263+0000","last_became_active":"2023-10-30T17:33:51.001524+0000","last_became_peered":"2023-10-30T17:33:51.001524+0000","last_unstale":"2023-10-30T17:55:25.744263+0000","last_undegraded":"2023-10-30T17:55:25.744263+0000","last_fullsized":"2023-10-30T17:55:25.744263+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:25.742821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:25.742821+0000","last_clean_scrub_stamp":"2023-10-30T17:55:25.742821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:29.684201+0000","scrub_duration":0.071310417000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":880,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:18.694788+0000","last_change":"2023-10-30T17:55:18.694788+0000","last_active":"2023-10-30T17:55:18.694788+0000","last_peered":"2023-10-30T17:55:18.694788+0000","last_clean":"2023-10-30T17:55:18.694788+0000","last_became_active":"2023-10-30T17:33:58.874661+0000","last_became_peered":"2023-10-30T17:33:58.874661+0000","last_unstale":"2023-10-30T17:55:18.694788+0000","last_undegraded":"2023-10-30T17:55:18.694788+0000","last_fullsized":"2023-10-30T17:55:18.694788+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:18.691215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:18.691215+0000","last_clean_scrub_stamp":"2023-10-30T17:55:18.691215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:51:38.631964+0000","scrub_duration":0.040631130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":860,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:27.285688+0000","last_change":"2023-10-30T17:55:27.286706+0000","last_active":"2023-10-30T17:55:27.285688+0000","last_peered":"2023-10-30T17:55:27.285688+0000","last_clean":"2023-10-30T17:55:27.285688+0000","last_became_active":"2023-10-30T17:33:39.086390+0000","last_became_peered":"2023-10-30T17:33:39.086390+0000","last_unstale":"2023-10-30T17:55:27.285688+0000","last_undegraded":"2023-10-30T17:55:27.285688+0000","last_fullsized":"2023-10-30T17:55:27.285688+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:27.284380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:27.284380+0000","last_clean_scrub_stamp":"2023-10-30T17:55:27.284380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:00:33.325612+0000","scrub_duration":0.046134963000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:17.576975+0000","last_change":"2023-10-30T17:55:17.578046+0000","last_active":"2023-10-30T17:55:17.576975+0000","last_peered":"2023-10-30T17:55:17.576975+0000","last_clean":"2023-10-30T17:55:17.576975+0000","last_became_active":"2023-10-30T17:33:39.965244+0000","last_became_peered":"2023-10-30T17:33:39.965244+0000","last_unstale":"2023-10-30T17:55:17.576975+0000","last_undegraded":"2023-10-30T17:55:17.576975+0000","last_fullsized":"2023-10-30T17:55:17.576975+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:17.575634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:17.575634+0000","last_clean_scrub_stamp":"2023-10-30T17:55:17.575634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:48.273182+0000","scrub_duration":0.042278923000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:23.456536+0000","last_change":"2023-10-30T17:55:23.457600+0000","last_active":"2023-10-30T17:55:23.456536+0000","last_peered":"2023-10-30T17:55:23.456536+0000","last_clean":"2023-10-30T17:55:23.456536+0000","last_became_active":"2023-10-30T17:33:46.562080+0000","last_became_peered":"2023-10-30T17:33:46.562080+0000","last_unstale":"2023-10-30T17:55:23.456536+0000","last_undegraded":"2023-10-30T17:55:23.456536+0000","last_fullsized":"2023-10-30T17:55:23.456536+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:23.455039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:23.455039+0000","last_clean_scrub_stamp":"2023-10-30T17:55:23.455039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:26:27.125563+0000","scrub_duration":0.072097618000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":879,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:12.512329+0000","last_change":"2023-10-30T17:55:12.513855+0000","last_active":"2023-10-30T17:55:12.512329+0000","last_peered":"2023-10-30T17:55:12.512329+0000","last_clean":"2023-10-30T17:55:12.512329+0000","last_became_active":"2023-10-30T17:33:40.798989+0000","last_became_peered":"2023-10-30T17:33:40.798989+0000","last_unstale":"2023-10-30T17:55:12.512329+0000","last_undegraded":"2023-10-30T17:55:12.512329+0000","last_fullsized":"2023-10-30T17:55:12.512329+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:12.510278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:12.510278+0000","last_clean_scrub_stamp":"2023-10-30T17:55:12.510278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:35.159349+0000","scrub_duration":0.041716329000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:21.357369+0000","last_change":"2023-10-30T17:55:21.358376+0000","last_active":"2023-10-30T17:55:21.357369+0000","last_peered":"2023-10-30T17:55:21.357369+0000","last_clean":"2023-10-30T17:55:21.357369+0000","last_became_active":"2023-10-30T17:32:35.367487+0000","last_became_peered":"2023-10-30T17:32:35.367487+0000","last_unstale":"2023-10-30T17:55:21.357369+0000","last_undegraded":"2023-10-30T17:55:21.357369+0000","last_fullsized":"2023-10-30T17:55:21.357369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:21.356030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:21.356030+0000","last_clean_scrub_stamp":"2023-10-30T17:55:21.356030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:04.756228+0000","scrub_duration":0.033632153999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":873,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:38.383989+0000","last_change":"2023-10-30T17:55:38.383989+0000","last_active":"2023-10-30T17:55:38.383989+0000","last_peered":"2023-10-30T17:55:38.383989+0000","last_clean":"2023-10-30T17:55:38.383989+0000","last_became_active":"2023-10-30T17:33:38.284317+0000","last_became_peered":"2023-10-30T17:33:38.284317+0000","last_unstale":"2023-10-30T17:55:38.383989+0000","last_undegraded":"2023-10-30T17:55:38.383989+0000","last_fullsized":"2023-10-30T17:55:38.383989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:38.381736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:38.381736+0000","last_clean_scrub_stamp":"2023-10-30T17:55:38.381736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:05.938980+0000","scrub_duration":0.031891652,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":877,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:08.573133+0000","last_change":"2023-10-30T17:55:08.574169+0000","last_active":"2023-10-30T17:55:08.573133+0000","last_peered":"2023-10-30T17:55:08.573133+0000","last_clean":"2023-10-30T17:55:08.573133+0000","last_became_active":"2023-10-30T17:32:34.557021+0000","last_became_peered":"2023-10-30T17:32:34.557021+0000","last_unstale":"2023-10-30T17:55:08.573133+0000","last_undegraded":"2023-10-30T17:55:08.573133+0000","last_fullsized":"2023-10-30T17:55:08.573133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:08.571819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:08.571819+0000","last_clean_scrub_stamp":"2023-10-30T17:55:08.571819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:38:46.258857+0000","scrub_duration":0.028442723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"914'8","reported_seq":875,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:28.321727+0000","last_change":"2023-10-30T17:55:28.323254+0000","last_active":"2023-10-30T17:55:28.321727+0000","last_peered":"2023-10-30T17:55:28.321727+0000","last_clean":"2023-10-30T17:55:28.321727+0000","last_became_active":"2023-10-30T17:33:39.211129+0000","last_became_peered":"2023-10-30T17:33:39.211129+0000","last_unstale":"2023-10-30T17:55:28.321727+0000","last_undegraded":"2023-10-30T17:55:28.321727+0000","last_fullsized":"2023-10-30T17:55:28.321727+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'8","last_scrub_stamp":"2023-10-30T17:55:28.320436+0000","last_deep_scrub":"914'8","last_deep_scrub_stamp":"2023-10-30T17:55:28.320436+0000","last_clean_scrub_stamp":"2023-10-30T17:55:28.320436+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:58.798076+0000","scrub_duration":0.10408546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:19.390591+0000","last_change":"2023-10-30T17:55:19.391631+0000","last_active":"2023-10-30T17:55:19.390591+0000","last_peered":"2023-10-30T17:55:19.390591+0000","last_clean":"2023-10-30T17:55:19.390591+0000","last_became_active":"2023-10-30T17:32:35.357489+0000","last_became_peered":"2023-10-30T17:32:35.357489+0000","last_unstale":"2023-10-30T17:55:19.390591+0000","last_undegraded":"2023-10-30T17:55:19.390591+0000","last_fullsized":"2023-10-30T17:55:19.390591+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:19.389304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:19.389304+0000","last_clean_scrub_stamp":"2023-10-30T17:55:19.389304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:18.913618+0000","scrub_duration":0.030571640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":869,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:06.175371+0000","last_change":"2023-10-30T17:55:06.176783+0000","last_active":"2023-10-30T17:55:06.175371+0000","last_peered":"2023-10-30T17:55:06.175371+0000","last_clean":"2023-10-30T17:55:06.175371+0000","last_became_active":"2023-10-30T17:33:40.584323+0000","last_became_peered":"2023-10-30T17:33:40.584323+0000","last_unstale":"2023-10-30T17:55:06.175371+0000","last_undegraded":"2023-10-30T17:55:06.175371+0000","last_fullsized":"2023-10-30T17:55:06.175371+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:06.173670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:06.173670+0000","last_clean_scrub_stamp":"2023-10-30T17:55:06.173670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:41.569418+0000","scrub_duration":0.033937040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":874,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:11.580393+0000","last_change":"2023-10-30T17:55:11.580393+0000","last_active":"2023-10-30T17:55:11.580393+0000","last_peered":"2023-10-30T17:55:11.580393+0000","last_clean":"2023-10-30T17:55:11.580393+0000","last_became_active":"2023-10-30T17:32:34.631494+0000","last_became_peered":"2023-10-30T17:32:34.631494+0000","last_unstale":"2023-10-30T17:55:11.580393+0000","last_undegraded":"2023-10-30T17:55:11.580393+0000","last_fullsized":"2023-10-30T17:55:11.580393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:11.576330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:11.576330+0000","last_clean_scrub_stamp":"2023-10-30T17:55:11.576330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:45:39.718092+0000","scrub_duration":0.043490681000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":872,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:37.391306+0000","last_change":"2023-10-30T17:55:37.392298+0000","last_active":"2023-10-30T17:55:37.391306+0000","last_peered":"2023-10-30T17:55:37.391306+0000","last_clean":"2023-10-30T17:55:37.391306+0000","last_became_active":"2023-10-30T17:33:46.588555+0000","last_became_peered":"2023-10-30T17:33:46.588555+0000","last_unstale":"2023-10-30T17:55:37.391306+0000","last_undegraded":"2023-10-30T17:55:37.391306+0000","last_fullsized":"2023-10-30T17:55:37.391306+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:37.389945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:37.389945+0000","last_clean_scrub_stamp":"2023-10-30T17:55:37.389945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:59:24.183403+0000","scrub_duration":0.054508319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:16.403101+0000","last_change":"2023-10-30T17:55:16.404127+0000","last_active":"2023-10-30T17:55:16.403101+0000","last_peered":"2023-10-30T17:55:16.403101+0000","last_clean":"2023-10-30T17:55:16.403101+0000","last_became_active":"2023-10-30T17:32:35.383414+0000","last_became_peered":"2023-10-30T17:32:35.383414+0000","last_unstale":"2023-10-30T17:55:16.403101+0000","last_undegraded":"2023-10-30T17:55:16.403101+0000","last_fullsized":"2023-10-30T17:55:16.403101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:16.401822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:16.401822+0000","last_clean_scrub_stamp":"2023-10-30T17:55:16.401822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:24.597562+0000","scrub_duration":0.028021751000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":885,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:17.420795+0000","last_change":"2023-10-30T17:55:17.421804+0000","last_active":"2023-10-30T17:55:17.420795+0000","last_peered":"2023-10-30T17:55:17.420795+0000","last_clean":"2023-10-30T17:55:17.420795+0000","last_became_active":"2023-10-30T17:33:38.176989+0000","last_became_peered":"2023-10-30T17:33:38.176989+0000","last_unstale":"2023-10-30T17:55:17.420795+0000","last_undegraded":"2023-10-30T17:55:17.420795+0000","last_fullsized":"2023-10-30T17:55:17.420795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:17.419519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:17.419519+0000","last_clean_scrub_stamp":"2023-10-30T17:55:17.419519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:04:02.018126+0000","scrub_duration":0.036305485999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":893,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:16.387462+0000","last_change":"2023-10-30T17:55:16.387462+0000","last_active":"2023-10-30T17:55:16.387462+0000","last_peered":"2023-10-30T17:55:16.387462+0000","last_clean":"2023-10-30T17:55:16.387462+0000","last_became_active":"2023-10-30T17:32:34.287031+0000","last_became_peered":"2023-10-30T17:32:34.287031+0000","last_unstale":"2023-10-30T17:55:16.387462+0000","last_undegraded":"2023-10-30T17:55:16.387462+0000","last_fullsized":"2023-10-30T17:55:16.387462+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:16.385069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:16.385069+0000","last_clean_scrub_stamp":"2023-10-30T17:55:16.385069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:04.642185+0000","scrub_duration":0.043308434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":885,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:21.487482+0000","last_change":"2023-10-30T17:55:21.488487+0000","last_active":"2023-10-30T17:55:21.487482+0000","last_peered":"2023-10-30T17:55:21.487482+0000","last_clean":"2023-10-30T17:55:21.487482+0000","last_became_active":"2023-10-30T17:33:41.116885+0000","last_became_peered":"2023-10-30T17:33:41.116885+0000","last_unstale":"2023-10-30T17:55:21.487482+0000","last_undegraded":"2023-10-30T17:55:21.487482+0000","last_fullsized":"2023-10-30T17:55:21.487482+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:21.486124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:21.486124+0000","last_clean_scrub_stamp":"2023-10-30T17:55:21.486124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:12.382300+0000","scrub_duration":0.040156958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":866,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:09.309575+0000","last_change":"2023-10-30T17:55:09.309575+0000","last_active":"2023-10-30T17:55:09.309575+0000","last_peered":"2023-10-30T17:55:09.309575+0000","last_clean":"2023-10-30T17:55:09.309575+0000","last_became_active":"2023-10-30T17:32:35.277163+0000","last_became_peered":"2023-10-30T17:32:35.277163+0000","last_unstale":"2023-10-30T17:55:09.309575+0000","last_undegraded":"2023-10-30T17:55:09.309575+0000","last_fullsized":"2023-10-30T17:55:09.309575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:09.302612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:09.302612+0000","last_clean_scrub_stamp":"2023-10-30T17:55:09.302612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:08:21.503163+0000","scrub_duration":0.044067580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":860,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:42.350313+0000","last_change":"2023-10-30T17:55:42.363984+0000","last_active":"2023-10-30T17:55:42.350313+0000","last_peered":"2023-10-30T17:55:42.350313+0000","last_clean":"2023-10-30T17:55:42.350313+0000","last_became_active":"2023-10-30T17:33:43.736021+0000","last_became_peered":"2023-10-30T17:33:43.736021+0000","last_unstale":"2023-10-30T17:55:42.350313+0000","last_undegraded":"2023-10-30T17:55:42.350313+0000","last_fullsized":"2023-10-30T17:55:42.350313+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:42.348958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:42.348958+0000","last_clean_scrub_stamp":"2023-10-30T17:55:42.348958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:39:42.742957+0000","scrub_duration":0.072589890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:23.334243+0000","last_change":"2023-10-30T17:55:23.335344+0000","last_active":"2023-10-30T17:55:23.334243+0000","last_peered":"2023-10-30T17:55:23.334243+0000","last_clean":"2023-10-30T17:55:23.334243+0000","last_became_active":"2023-10-30T17:32:35.623395+0000","last_became_peered":"2023-10-30T17:32:35.623395+0000","last_unstale":"2023-10-30T17:55:23.334243+0000","last_undegraded":"2023-10-30T17:55:23.334243+0000","last_fullsized":"2023-10-30T17:55:23.334243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:23.332803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:23.332803+0000","last_clean_scrub_stamp":"2023-10-30T17:55:23.332803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T05:07:30.217748+0000","scrub_duration":0.035208454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"914'78","reported_seq":978,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:10.218608+0000","last_change":"2023-10-30T17:55:10.220901+0000","last_active":"2023-10-30T17:55:10.218608+0000","last_peered":"2023-10-30T17:55:10.218608+0000","last_clean":"2023-10-30T17:55:10.218608+0000","last_became_active":"2023-10-30T17:32:28.686958+0000","last_became_peered":"2023-10-30T17:32:28.686958+0000","last_unstale":"2023-10-30T17:55:10.218608+0000","last_undegraded":"2023-10-30T17:55:10.218608+0000","last_fullsized":"2023-10-30T17:55:10.218608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'78","last_scrub_stamp":"2023-10-30T17:55:10.216301+0000","last_deep_scrub":"914'78","last_deep_scrub_stamp":"2023-10-30T17:55:10.216301+0000","last_clean_scrub_stamp":"2023-10-30T17:55:10.216301+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:16:00.152248+0000","scrub_duration":0.12796829700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":869,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:07.218625+0000","last_change":"2023-10-30T17:55:07.219619+0000","last_active":"2023-10-30T17:55:07.218625+0000","last_peered":"2023-10-30T17:55:07.218625+0000","last_clean":"2023-10-30T17:55:07.218625+0000","last_became_active":"2023-10-30T17:33:40.602436+0000","last_became_peered":"2023-10-30T17:33:40.602436+0000","last_unstale":"2023-10-30T17:55:07.218625+0000","last_undegraded":"2023-10-30T17:55:07.218625+0000","last_fullsized":"2023-10-30T17:55:07.218625+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:07.217190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:07.217190+0000","last_clean_scrub_stamp":"2023-10-30T17:55:07.217190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:25.972202+0000","scrub_duration":0.033129934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":860,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:40.335631+0000","last_change":"2023-10-30T17:55:40.336633+0000","last_active":"2023-10-30T17:55:40.335631+0000","last_peered":"2023-10-30T17:55:40.335631+0000","last_clean":"2023-10-30T17:55:40.335631+0000","last_became_active":"2023-10-30T17:33:38.616054+0000","last_became_peered":"2023-10-30T17:33:38.616054+0000","last_unstale":"2023-10-30T17:55:40.335631+0000","last_undegraded":"2023-10-30T17:55:40.335631+0000","last_fullsized":"2023-10-30T17:55:40.335631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:40.334343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:40.334343+0000","last_clean_scrub_stamp":"2023-10-30T17:55:40.334343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:49:29.928538+0000","scrub_duration":0.031224485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":864,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:19.186826+0000","last_change":"2023-10-30T17:55:19.187904+0000","last_active":"2023-10-30T17:55:19.186826+0000","last_peered":"2023-10-30T17:55:19.186826+0000","last_clean":"2023-10-30T17:55:19.186826+0000","last_became_active":"2023-10-30T17:33:40.735511+0000","last_became_peered":"2023-10-30T17:33:40.735511+0000","last_unstale":"2023-10-30T17:55:19.186826+0000","last_undegraded":"2023-10-30T17:55:19.186826+0000","last_fullsized":"2023-10-30T17:55:19.186826+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:19.185432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:19.185432+0000","last_clean_scrub_stamp":"2023-10-30T17:55:19.185432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:17.820548+0000","scrub_duration":0.067969350999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":867,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:14.589320+0000","last_change":"2023-10-30T17:55:14.590436+0000","last_active":"2023-10-30T17:55:14.589320+0000","last_peered":"2023-10-30T17:55:14.589320+0000","last_clean":"2023-10-30T17:55:14.589320+0000","last_became_active":"2023-10-30T17:33:39.228561+0000","last_became_peered":"2023-10-30T17:33:39.228561+0000","last_unstale":"2023-10-30T17:55:14.589320+0000","last_undegraded":"2023-10-30T17:55:14.589320+0000","last_fullsized":"2023-10-30T17:55:14.589320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:14.587906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:14.587906+0000","last_clean_scrub_stamp":"2023-10-30T17:55:14.587906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:23:56.357532+0000","scrub_duration":0.056479152999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":884,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:20.480379+0000","last_change":"2023-10-30T17:55:20.481377+0000","last_active":"2023-10-30T17:55:20.480379+0000","last_peered":"2023-10-30T17:55:20.480379+0000","last_clean":"2023-10-30T17:55:20.480379+0000","last_became_active":"2023-10-30T17:33:41.605208+0000","last_became_peered":"2023-10-30T17:33:41.605208+0000","last_unstale":"2023-10-30T17:55:20.480379+0000","last_undegraded":"2023-10-30T17:55:20.480379+0000","last_fullsized":"2023-10-30T17:55:20.480379+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:20.479126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:20.479126+0000","last_clean_scrub_stamp":"2023-10-30T17:55:20.479126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:49:32.198607+0000","scrub_duration":0.066395858000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":880,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:13.534354+0000","last_change":"2023-10-30T17:55:13.534354+0000","last_active":"2023-10-30T17:55:13.534354+0000","last_peered":"2023-10-30T17:55:13.534354+0000","last_clean":"2023-10-30T17:55:13.534354+0000","last_became_active":"2023-10-30T17:33:44.434104+0000","last_became_peered":"2023-10-30T17:33:44.434104+0000","last_unstale":"2023-10-30T17:55:13.534354+0000","last_undegraded":"2023-10-30T17:55:13.534354+0000","last_fullsized":"2023-10-30T17:55:13.534354+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:13.528576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:13.528576+0000","last_clean_scrub_stamp":"2023-10-30T17:55:13.528576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:35:11.000117+0000","scrub_duration":0.031392597000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":868,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:12.587925+0000","last_change":"2023-10-30T17:55:12.589027+0000","last_active":"2023-10-30T17:55:12.587925+0000","last_peered":"2023-10-30T17:55:12.587925+0000","last_clean":"2023-10-30T17:55:12.587925+0000","last_became_active":"2023-10-30T17:33:43.322860+0000","last_became_peered":"2023-10-30T17:33:43.322860+0000","last_unstale":"2023-10-30T17:55:12.587925+0000","last_undegraded":"2023-10-30T17:55:12.587925+0000","last_fullsized":"2023-10-30T17:55:12.587925+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:12.586555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:12.586555+0000","last_clean_scrub_stamp":"2023-10-30T17:55:12.586555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:26.081277+0000","scrub_duration":0.065328111999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":880,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:11.450149+0000","last_change":"2023-10-30T17:55:11.450149+0000","last_active":"2023-10-30T17:55:11.450149+0000","last_peered":"2023-10-30T17:55:11.450149+0000","last_clean":"2023-10-30T17:55:11.450149+0000","last_became_active":"2023-10-30T17:33:53.130615+0000","last_became_peered":"2023-10-30T17:33:53.130615+0000","last_unstale":"2023-10-30T17:55:11.450149+0000","last_undegraded":"2023-10-30T17:55:11.450149+0000","last_fullsized":"2023-10-30T17:55:11.450149+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:11.447787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:11.447787+0000","last_clean_scrub_stamp":"2023-10-30T17:55:11.447787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:15:08.444731+0000","scrub_duration":0.02798376,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":865,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:13.182186+0000","last_change":"2023-10-30T17:55:13.182186+0000","last_active":"2023-10-30T17:55:13.182186+0000","last_peered":"2023-10-30T17:55:13.182186+0000","last_clean":"2023-10-30T17:55:13.182186+0000","last_became_active":"2023-10-30T17:33:37.540749+0000","last_became_peered":"2023-10-30T17:33:37.540749+0000","last_unstale":"2023-10-30T17:55:13.182186+0000","last_undegraded":"2023-10-30T17:55:13.182186+0000","last_fullsized":"2023-10-30T17:55:13.182186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:13.179784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:13.179784+0000","last_clean_scrub_stamp":"2023-10-30T17:55:13.179784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:38.127850+0000","scrub_duration":0.071081134000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":860,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:34.297846+0000","last_change":"2023-10-30T17:55:34.298846+0000","last_active":"2023-10-30T17:55:34.297846+0000","last_peered":"2023-10-30T17:55:34.297846+0000","last_clean":"2023-10-30T17:55:34.297846+0000","last_became_active":"2023-10-30T17:33:58.383320+0000","last_became_peered":"2023-10-30T17:33:58.383320+0000","last_unstale":"2023-10-30T17:55:34.297846+0000","last_undegraded":"2023-10-30T17:55:34.297846+0000","last_fullsized":"2023-10-30T17:55:34.297846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:34.296543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:34.296543+0000","last_clean_scrub_stamp":"2023-10-30T17:55:34.296543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:02:04.049740+0000","scrub_duration":0.033278672000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":870,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:19.494465+0000","last_change":"2023-10-30T17:55:19.494465+0000","last_active":"2023-10-30T17:55:19.494465+0000","last_peered":"2023-10-30T17:55:19.494465+0000","last_clean":"2023-10-30T17:55:19.494465+0000","last_became_active":"2023-10-30T17:33:38.589479+0000","last_became_peered":"2023-10-30T17:33:38.589479+0000","last_unstale":"2023-10-30T17:55:19.494465+0000","last_undegraded":"2023-10-30T17:55:19.494465+0000","last_fullsized":"2023-10-30T17:55:19.494465+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:19.489037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:19.489037+0000","last_clean_scrub_stamp":"2023-10-30T17:55:19.489037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:46.197528+0000","scrub_duration":0.068705826999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:21.428804+0000","last_change":"2023-10-30T17:55:21.429918+0000","last_active":"2023-10-30T17:55:21.428804+0000","last_peered":"2023-10-30T17:55:21.428804+0000","last_clean":"2023-10-30T17:55:21.428804+0000","last_became_active":"2023-10-30T17:33:40.476315+0000","last_became_peered":"2023-10-30T17:33:40.476315+0000","last_unstale":"2023-10-30T17:55:21.428804+0000","last_undegraded":"2023-10-30T17:55:21.428804+0000","last_fullsized":"2023-10-30T17:55:21.428804+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:21.427013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:21.427013+0000","last_clean_scrub_stamp":"2023-10-30T17:55:21.427013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:15.296878+0000","scrub_duration":0.036816092000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":860,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-10-30T17:55:31.318338+0000","last_change":"2023-10-30T17:55:31.320734+0000","last_active":"2023-10-30T17:55:31.318338+0000","last_peered":"2023-10-30T17:55:31.318338+0000","last_clean":"2023-10-30T17:55:31.318338+0000","last_became_active":"2023-10-30T17:33:38.795801+0000","last_became_peered":"2023-10-30T17:33:38.795801+0000","last_unstale":"2023-10-30T17:55:31.318338+0000","last_undegraded":"2023-10-30T17:55:31.318338+0000","last_fullsized":"2023-10-30T17:55:31.318338+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T17:55:31.315183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T17:55:31.315183+0000","last_clean_scrub_stamp":"2023-10-30T17:55:31.315183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:56:03.205094+0000","scrub_duration":0.055452492999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":17,"seq":73014444312,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60908,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310932,"statfs":{"total":96636764160,"available":96574394368,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 17:55:23 2023","interfaces":[{"interface":"back","average":{"1min":12.914,"5min":14.653,"15min":19.036999999999999},"min":{"1min":5.3259999999999996,"5min":4.2460000000000004,"15min":4.2460000000000004},"max":{"1min":63.363,"5min":217.97399999999999,"15min":296.49400000000003},"last":8.9689999999999994},{"interface":"front","average":{"1min":10.414,"5min":15.103999999999999,"15min":19.773},"min":{"1min":5.5309999999999997,"5min":5.5309999999999997,"15min":4.6299999999999999},"max":{"1min":25.385000000000002,"5min":134.833,"15min":276.49799999999999},"last":9.7379999999999995}]},{"osd":2,"last update":"Mon Oct 30 17:55:20 2023","interfaces":[{"interface":"back","average":{"1min":14.949999999999999,"5min":16.766999999999999,"15min":23.184999999999999},"min":{"1min":8.1669999999999998,"5min":6.319,"15min":5.915},"max":{"1min":33.576999999999998,"5min":120.46299999999999,"15min":591.02499999999998},"last":15.960000000000001},{"interface":"front","average":{"1min":15.061,"5min":18.091000000000001,"15min":23.324000000000002},"min":{"1min":9.3659999999999997,"5min":7.6520000000000001,"15min":6.4930000000000003},"max":{"1min":70.671000000000006,"5min":129.03299999999999,"15min":259.52300000000002},"last":12.566000000000001}]},{"osd":3,"last update":"Mon Oct 30 17:54:58 2023","interfaces":[{"interface":"back","average":{"1min":13.741,"5min":75.924000000000007,"15min":40.366999999999997},"min":{"1min":6.4290000000000003,"5min":6.4290000000000003,"15min":5.8099999999999996},"max":{"1min":62.359000000000002,"5min":5036.1790000000001,"15min":5036.1790000000001},"last":7.4729999999999999},{"interface":"front","average":{"1min":17.495999999999999,"5min":79.316999999999993,"15min":44.966999999999999},"min":{"1min":7.6950000000000003,"5min":7.2770000000000001,"15min":6.4909999999999997},"max":{"1min":47.174999999999997,"5min":5057.1229999999996,"15min":5057.1229999999996},"last":14.65}]},{"osd":4,"last update":"Mon Oct 30 17:55:23 2023","interfaces":[{"interface":"back","average":{"1min":15.284000000000001,"5min":19.625,"15min":21.283000000000001},"min":{"1min":7.5380000000000003,"5min":7.3259999999999996,"15min":6.3689999999999998},"max":{"1min":64.394000000000005,"5min":159.94399999999999,"15min":281.16399999999999},"last":8.2460000000000004},{"interface":"front","average":{"1min":17.460000000000001,"5min":23.212,"15min":26},"min":{"1min":8.9160000000000004,"5min":8.9160000000000004,"15min":5.8879999999999999},"max":{"1min":49.860999999999997,"5min":198.04900000000001,"15min":218.85300000000001},"last":17.518000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:55:23 2023","interfaces":[{"interface":"back","average":{"1min":17.294,"5min":22.064,"15min":26.204000000000001},"min":{"1min":8.1850000000000005,"5min":8.1850000000000005,"15min":7.9370000000000003},"max":{"1min":80.078000000000003,"5min":260.81099999999998,"15min":260.81099999999998},"last":13.250999999999999},{"interface":"front","average":{"1min":18.815000000000001,"5min":20.047999999999998,"15min":24.436},"min":{"1min":8.5180000000000007,"5min":7.6029999999999998,"15min":7.1189999999999998},"max":{"1min":83.149000000000001,"5min":155.02000000000001,"15min":236.83500000000001},"last":10.446}]},{"osd":6,"last update":"Mon Oct 30 17:55:23 2023","interfaces":[{"interface":"back","average":{"1min":19.027000000000001,"5min":20.858000000000001,"15min":24.974},"min":{"1min":8.2159999999999993,"5min":8.2159999999999993,"15min":8.2159999999999993},"max":{"1min":80.769999999999996,"5min":125.158,"15min":246.01499999999999},"last":11.186},{"interface":"front","average":{"1min":18.800999999999998,"5min":22.594999999999999,"15min":26.859999999999999},"min":{"1min":10.052,"5min":6.8449999999999998,"15min":6.8449999999999998},"max":{"1min":79.447000000000003,"5min":189.01599999999999,"15min":191.96700000000001},"last":13.901999999999999}]},{"osd":7,"last update":"Mon Oct 30 17:55:20 2023","interfaces":[{"interface":"back","average":{"1min":19.669,"5min":22.777999999999999,"15min":27.725000000000001},"min":{"1min":11.65,"5min":10.622999999999999,"15min":9.157},"max":{"1min":81.537000000000006,"5min":135.47800000000001,"15min":262.74200000000002},"last":15.329000000000001},{"interface":"front","average":{"1min":22.597999999999999,"5min":27.381,"15min":31.349},"min":{"1min":12.738,"5min":12.188000000000001,"15min":10.122999999999999},"max":{"1min":84.094999999999999,"5min":140.83500000000001,"15min":264.81},"last":18.411999999999999}]}]},{"osd":3,"up_from":16,"seq":68719477008,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83120,"kb_used_data":7528,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288720,"statfs":{"total":96636764160,"available":96551649280,"internally_reserved":0,"allocated":7708672,"data_stored":4236976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":30.98,"5min":20.527999999999999,"15min":16.279},"min":{"1min":5.5099999999999998,"5min":4.4489999999999998,"15min":4.3869999999999996},"max":{"1min":184.12899999999999,"5min":203.85499999999999,"15min":203.85499999999999},"last":9.2110000000000003},{"interface":"front","average":{"1min":33.973999999999997,"5min":21.707999999999998,"15min":18.739999999999998},"min":{"1min":6.4829999999999997,"5min":4.8449999999999998,"15min":4.367},"max":{"1min":138.00399999999999,"5min":176.83500000000001,"15min":364.05399999999997},"last":16.122}]},{"osd":1,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":20.564,"5min":23.222000000000001,"15min":23.289000000000001},"min":{"1min":7.4489999999999998,"5min":7.1909999999999998,"15min":5.5380000000000003},"max":{"1min":124.999,"5min":178.04300000000001,"15min":190.55199999999999},"last":15.372999999999999},{"interface":"front","average":{"1min":19.936,"5min":23.527000000000001,"15min":22.817},"min":{"1min":7.7690000000000001,"5min":5.9290000000000003,"15min":5.609},"max":{"1min":105.627,"5min":210.381,"15min":240.64500000000001},"last":7.2439999999999998}]},{"osd":2,"last update":"Mon Oct 30 17:55:21 2023","interfaces":[{"interface":"back","average":{"1min":24.437000000000001,"5min":29.425000000000001,"15min":23.103999999999999},"min":{"1min":8.4120000000000008,"5min":7.2969999999999997,"15min":5.234},"max":{"1min":116.946,"5min":251.941,"15min":397.50099999999998},"last":16.864000000000001},{"interface":"front","average":{"1min":14.188000000000001,"5min":21.253,"15min":23},"min":{"1min":6.5750000000000002,"5min":6.54,"15min":6.54},"max":{"1min":68.379999999999995,"5min":147.50399999999999,"15min":219.65000000000001},"last":8.2699999999999996}]},{"osd":4,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":24.489999999999998,"5min":34.338000000000001,"15min":26.545000000000002},"min":{"1min":9.0999999999999996,"5min":7.7729999999999997,"15min":5.8140000000000001},"max":{"1min":126.723,"5min":318.92200000000003,"15min":318.92200000000003},"last":9.8040000000000003},{"interface":"front","average":{"1min":26.800000000000001,"5min":27.832999999999998,"15min":21.242999999999999},"min":{"1min":8.5909999999999993,"5min":6.2450000000000001,"15min":6.2450000000000001},"max":{"1min":137.06999999999999,"5min":268.01600000000002,"15min":268.01600000000002},"last":11.081}]},{"osd":5,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":43.463000000000001,"5min":26.356000000000002,"15min":23.449000000000002},"min":{"1min":10.387,"5min":7.7569999999999997,"15min":7.7569999999999997},"max":{"1min":217.596,"5min":217.596,"15min":221.821},"last":7.9960000000000004},{"interface":"front","average":{"1min":35.868000000000002,"5min":30.478999999999999,"15min":26.512},"min":{"1min":9.6460000000000008,"5min":7.2009999999999996,"15min":7.2009999999999996},"max":{"1min":223.203,"5min":225.15299999999999,"15min":225.15299999999999},"last":11.711}]},{"osd":6,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":33.683999999999997,"5min":31.859000000000002,"15min":26.734000000000002},"min":{"1min":11.108000000000001,"5min":7.9589999999999996,"15min":7.7439999999999998},"max":{"1min":137.48599999999999,"5min":272.536,"15min":272.536},"last":10.496},{"interface":"front","average":{"1min":30.719000000000001,"5min":24.899999999999999,"15min":23.899000000000001},"min":{"1min":9.8620000000000001,"5min":9.3040000000000003,"15min":7.524},"max":{"1min":121.95999999999999,"5min":162.61799999999999,"15min":199.892},"last":12.301}]},{"osd":7,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":28.047000000000001,"5min":30.155000000000001,"15min":28.039000000000001},"min":{"1min":12.372999999999999,"5min":9.8399999999999999,"15min":9.7059999999999995},"max":{"1min":124.315,"5min":196.05699999999999,"15min":196.66399999999999},"last":13.178000000000001},{"interface":"front","average":{"1min":25.338000000000001,"5min":29.327999999999999,"15min":28.225000000000001},"min":{"1min":8.7590000000000003,"5min":8.7590000000000003,"15min":8.7590000000000003},"max":{"1min":157.10499999999999,"5min":182.07499999999999,"15min":221.59100000000001},"last":17.702000000000002}]}]},{"osd":1,"up_from":16,"seq":68719477019,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61300,"kb_used_data":7916,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310540,"statfs":{"total":96636764160,"available":96573992960,"internally_reserved":0,"allocated":8105984,"data_stored":4634811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:17 2023","interfaces":[{"interface":"back","average":{"1min":9.6950000000000003,"5min":12.606,"15min":16.495999999999999},"min":{"1min":5.2350000000000003,"5min":5.0149999999999997,"15min":3.8279999999999998},"max":{"1min":16.949000000000002,"5min":162.364,"15min":313.69400000000002},"last":5.2699999999999996},{"interface":"front","average":{"1min":12.805,"5min":13.266,"15min":16.631},"min":{"1min":7.0709999999999997,"5min":5.9829999999999997,"15min":4.9089999999999998},"max":{"1min":37.804000000000002,"5min":59.277000000000001,"15min":209.62799999999999},"last":11.619999999999999}]},{"osd":2,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":15.744999999999999,"15min":19.428999999999998},"min":{"1min":9.1430000000000007,"5min":6.6139999999999999,"15min":6.0279999999999996},"max":{"1min":48.945,"5min":84.930999999999997,"15min":267.83499999999998},"last":13.973000000000001},{"interface":"front","average":{"1min":18.347000000000001,"5min":18.187000000000001,"15min":18.768000000000001},"min":{"1min":10.164,"5min":7.0270000000000001,"15min":5.8550000000000004},"max":{"1min":60.828000000000003,"5min":89.875,"15min":178.44},"last":15.414}]},{"osd":3,"last update":"Mon Oct 30 17:55:20 2023","interfaces":[{"interface":"back","average":{"1min":12.201000000000001,"5min":246.72800000000001,"15min":93.194000000000003},"min":{"1min":5.5549999999999997,"5min":5.5549999999999997,"15min":5.2990000000000004},"max":{"1min":68.070999999999998,"5min":8788.2559999999994,"15min":8788.2559999999994},"last":7.2569999999999997},{"interface":"front","average":{"1min":17.672999999999998,"5min":247.571,"15min":96.239999999999995},"min":{"1min":10.819000000000001,"5min":10.159000000000001,"15min":6.8819999999999997},"max":{"1min":41.433999999999997,"5min":8677.7369999999992,"15min":8677.7369999999992},"last":16.149999999999999}]},{"osd":4,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":15.423999999999999,"5min":16.965,"15min":21.763999999999999},"min":{"1min":7.0250000000000004,"5min":6.8220000000000001,"15min":6.8220000000000001},"max":{"1min":49.585999999999999,"5min":91.501999999999995,"15min":235.125},"last":9.1519999999999992},{"interface":"front","average":{"1min":26.652999999999999,"5min":22.824000000000002,"15min":23.303999999999998},"min":{"1min":11.677,"5min":10.278,"15min":7.673},"max":{"1min":118.545,"5min":118.545,"15min":334.19900000000001},"last":12.206}]},{"osd":5,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":25.949999999999999,"5min":19.481999999999999,"15min":25.600000000000001},"min":{"1min":10.999000000000001,"5min":9.0389999999999997,"15min":7.6360000000000001},"max":{"1min":93.442999999999998,"5min":93.442999999999998,"15min":181.976},"last":11.119999999999999},{"interface":"front","average":{"1min":24.221,"5min":21.082999999999998,"15min":28.675999999999998},"min":{"1min":12.029,"5min":9.3759999999999994,"15min":8.0869999999999997},"max":{"1min":122.42700000000001,"5min":122.42700000000001,"15min":262.61799999999999},"last":17.029}]},{"osd":6,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":24.259,"5min":17.661999999999999,"15min":22.719999999999999},"min":{"1min":6.2750000000000004,"5min":6.2750000000000004,"15min":6.2750000000000004},"max":{"1min":173.261,"5min":200.369,"15min":249.25299999999999},"last":12.65},{"interface":"front","average":{"1min":25.248999999999999,"5min":21.978999999999999,"15min":19.956},"min":{"1min":13.238,"5min":9.452,"15min":6.2880000000000003},"max":{"1min":112.628,"5min":201.756,"15min":240.72200000000001},"last":17.827000000000002}]},{"osd":7,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":25.876999999999999,"5min":22.286999999999999,"15min":25.803999999999998},"min":{"1min":10.916,"5min":9.8200000000000003,"15min":9.4269999999999996},"max":{"1min":123.173,"5min":123.173,"15min":145.078},"last":18.600000000000001},{"interface":"front","average":{"1min":21.436,"5min":23.945,"15min":28.402000000000001},"min":{"1min":14.316000000000001,"5min":11.098000000000001,"15min":10.628},"max":{"1min":39.271999999999998,"5min":203.929,"15min":203.929},"last":13.382}]}]},{"osd":2,"up_from":15,"seq":64424509722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60908,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310932,"statfs":{"total":96636764160,"available":96574394368,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.643000000000001,"5min":13.712,"15min":16.542000000000002},"min":{"1min":5.9649999999999999,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":87.680999999999997,"5min":106.839,"15min":292.57299999999998},"last":7.2069999999999999},{"interface":"front","average":{"1min":16.539999999999999,"5min":17.335999999999999,"15min":20.198},"min":{"1min":7.569,"5min":7.1319999999999997,"15min":6.7489999999999997},"max":{"1min":88.846000000000004,"5min":111.07599999999999,"15min":283.46899999999999},"last":8.8629999999999995}]},{"osd":1,"last update":"Mon Oct 30 17:55:16 2023","interfaces":[{"interface":"back","average":{"1min":15.481999999999999,"5min":20.122,"15min":22.815999999999999},"min":{"1min":10.173,"5min":6.1539999999999999,"15min":6.1539999999999999},"max":{"1min":31.516999999999999,"5min":152.304,"15min":273.93599999999998},"last":12.125},{"interface":"front","average":{"1min":10.638,"5min":13.188000000000001,"15min":14.933},"min":{"1min":5.2080000000000002,"5min":4.9509999999999996,"15min":4.1890000000000001},"max":{"1min":24.187000000000001,"5min":109.767,"15min":152.87299999999999},"last":5.085}]},{"osd":3,"last update":"Mon Oct 30 17:55:16 2023","interfaces":[{"interface":"back","average":{"1min":15.285,"5min":220.51400000000001,"15min":84.436000000000007},"min":{"1min":7.0019999999999998,"5min":5.6429999999999998,"15min":5.6429999999999998},"max":{"1min":108.666,"5min":8749.8639999999996,"15min":8749.8639999999996},"last":7.8319999999999999},{"interface":"front","average":{"1min":21.670000000000002,"5min":226.71199999999999,"15min":90.552999999999997},"min":{"1min":11.273999999999999,"5min":7.7859999999999996,"15min":6.6799999999999997},"max":{"1min":109.779,"5min":8752.7450000000008,"15min":8752.7450000000008},"last":10.363}]},{"osd":4,"last update":"Mon Oct 30 17:55:11 2023","interfaces":[{"interface":"back","average":{"1min":14.714,"5min":20.983000000000001,"15min":25.411000000000001},"min":{"1min":8.6059999999999999,"5min":8.5990000000000002,"15min":8.5990000000000002},"max":{"1min":36.32,"5min":124.502,"15min":301.34399999999999},"last":16.419},{"interface":"front","average":{"1min":18.843,"5min":26.375,"15min":30.210999999999999},"min":{"1min":10.959,"5min":10.959,"15min":10.959},"max":{"1min":36.787999999999997,"5min":134.44800000000001,"15min":300.68900000000002},"last":17.335999999999999}]},{"osd":5,"last update":"Mon Oct 30 17:55:16 2023","interfaces":[{"interface":"back","average":{"1min":17.263000000000002,"5min":18.863,"15min":27.036000000000001},"min":{"1min":6.516,"5min":6.4249999999999998,"15min":6.4249999999999998},"max":{"1min":58.253999999999998,"5min":198.46700000000001,"15min":318.01900000000001},"last":13.356},{"interface":"front","average":{"1min":24.870000000000001,"5min":20.832999999999998,"15min":21.594999999999999},"min":{"1min":10.16,"5min":8.4760000000000009,"15min":5.8920000000000003},"max":{"1min":107.361,"5min":122.861,"15min":325.87099999999998},"last":14.718999999999999}]},{"osd":6,"last update":"Mon Oct 30 17:55:11 2023","interfaces":[{"interface":"back","average":{"1min":16.164999999999999,"5min":18.206,"15min":18.603000000000002},"min":{"1min":8.9909999999999997,"5min":8.7420000000000009,"15min":6.3979999999999997},"max":{"1min":37.514000000000003,"5min":70.927999999999997,"15min":176.274},"last":11.502000000000001},{"interface":"front","average":{"1min":19.661000000000001,"5min":24.739000000000001,"15min":28.327999999999999},"min":{"1min":9.016,"5min":9.016,"15min":9.016},"max":{"1min":39.222000000000001,"5min":133.02199999999999,"15min":306.81799999999998},"last":18.138000000000002}]},{"osd":7,"last update":"Mon Oct 30 17:55:11 2023","interfaces":[{"interface":"back","average":{"1min":12.196,"5min":15.977,"15min":20.768000000000001},"min":{"1min":8.3610000000000007,"5min":7.2549999999999999,"15min":6.9669999999999996},"max":{"1min":16.366,"5min":127.804,"15min":307.69499999999999},"last":9.7189999999999994},{"interface":"front","average":{"1min":17.422000000000001,"5min":21.34,"15min":26.643000000000001},"min":{"1min":10.691000000000001,"5min":8.9550000000000001,"15min":8.5700000000000003},"max":{"1min":28.623000000000001,"5min":129.26599999999999,"15min":317.13400000000001},"last":13.999000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542425,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83120,"kb_used_data":7528,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94288720,"statfs":{"total":96636764160,"available":96551649280,"internally_reserved":0,"allocated":7708672,"data_stored":4236976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77396427},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":17.451000000000001,"5min":16.484999999999999,"15min":16.552},"min":{"1min":5.1619999999999999,"5min":4.6230000000000002,"15min":4.1580000000000004},"max":{"1min":154.50899999999999,"5min":172.93199999999999,"15min":256.154},"last":16.405000000000001},{"interface":"front","average":{"1min":16.391999999999999,"5min":17.413,"15min":16.408999999999999},"min":{"1min":5.2610000000000001,"5min":4.5999999999999996,"15min":4.3959999999999999},"max":{"1min":144.61000000000001,"5min":176.59399999999999,"15min":258.25799999999998},"last":8.3539999999999992}]},{"osd":1,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":12.489000000000001,"5min":14.763999999999999,"15min":16.366},"min":{"1min":6.2990000000000004,"5min":4.9640000000000004,"15min":4.2939999999999996},"max":{"1min":26.224,"5min":83.355000000000004,"15min":175.297},"last":7.2729999999999997},{"interface":"front","average":{"1min":22.309000000000001,"5min":18.849,"15min":21.831},"min":{"1min":8.2100000000000009,"5min":8.0640000000000001,"15min":5.0439999999999996},"max":{"1min":75.745000000000005,"5min":139.392,"15min":334.67599999999999},"last":12.923999999999999}]},{"osd":2,"last update":"Mon Oct 30 17:55:05 2023","interfaces":[{"interface":"back","average":{"1min":20.262,"5min":18.870999999999999,"15min":20.088000000000001},"min":{"1min":7.7910000000000004,"5min":6.3410000000000002,"15min":6.3140000000000001},"max":{"1min":145.50800000000001,"5min":145.50800000000001,"15min":403.90899999999999},"last":9.9589999999999996},{"interface":"front","average":{"1min":26.34,"5min":26.309999999999999,"15min":25.690999999999999},"min":{"1min":9.641,"5min":8.4559999999999995,"15min":7.6829999999999998},"max":{"1min":146.65100000000001,"5min":276.67700000000002,"15min":354.34800000000001},"last":15.634}]},{"osd":3,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":24.350000000000001,"5min":166.03899999999999,"15min":69.540999999999997},"min":{"1min":8.3469999999999995,"5min":8.3469999999999995,"15min":6.8849999999999998},"max":{"1min":108.26600000000001,"5min":8740.9120000000003,"15min":8740.9120000000003},"last":11.327},{"interface":"front","average":{"1min":22.797000000000001,"5min":163.64400000000001,"15min":67.997},"min":{"1min":7.6699999999999999,"5min":7.3840000000000003,"15min":6.1120000000000001},"max":{"1min":109.111,"5min":8827.2800000000007,"15min":8827.2800000000007},"last":14.308999999999999}]},{"osd":4,"last update":"Mon Oct 30 17:55:12 2023","interfaces":[{"interface":"back","average":{"1min":25.326000000000001,"5min":21.077000000000002,"15min":23.256},"min":{"1min":7.4109999999999996,"5min":6.758,"15min":6.758},"max":{"1min":107.45399999999999,"5min":108.28100000000001,"15min":262.92899999999997},"last":18.283000000000001},{"interface":"front","average":{"1min":28.277000000000001,"5min":27.474,"15min":25.882000000000001},"min":{"1min":9.4670000000000005,"5min":9.4670000000000005,"15min":8.5839999999999996},"max":{"1min":112.31100000000001,"5min":204.27799999999999,"15min":297.99900000000002},"last":15.000999999999999}]},{"osd":5,"last update":"Mon Oct 30 17:55:05 2023","interfaces":[{"interface":"back","average":{"1min":28.478000000000002,"5min":24.763999999999999,"15min":27.021999999999998},"min":{"1min":11.518000000000001,"5min":9.4339999999999993,"15min":7.7729999999999997},"max":{"1min":173.75999999999999,"5min":235.08500000000001,"15min":298.197},"last":19.239000000000001},{"interface":"front","average":{"1min":19.510000000000002,"5min":21.577000000000002,"15min":21.898},"min":{"1min":10.781000000000001,"5min":6.5010000000000003,"15min":5.5010000000000003},"max":{"1min":79.013000000000005,"5min":266.30200000000002,"15min":280.34100000000001},"last":13.641}]},{"osd":6,"last update":"Mon Oct 30 17:55:05 2023","interfaces":[{"interface":"back","average":{"1min":26.076000000000001,"5min":23.91,"15min":23.134},"min":{"1min":8.6750000000000007,"5min":8.6750000000000007,"15min":6.9400000000000004},"max":{"1min":122.34099999999999,"5min":174.517,"15min":260.10399999999998},"last":12.253},{"interface":"front","average":{"1min":25.681999999999999,"5min":26.704999999999998,"15min":26.986999999999998},"min":{"1min":8.7430000000000003,"5min":8.7430000000000003,"15min":8.0060000000000002},"max":{"1min":79.855000000000004,"5min":140.06399999999999,"15min":257.62599999999998},"last":20.047999999999998}]}]},{"osd":6,"up_from":14,"seq":60129542428,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78956,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292884,"statfs":{"total":96636764160,"available":96555913216,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":12.414999999999999,"5min":15.685,"15min":16.145},"min":{"1min":4.617,"5min":4.3339999999999996,"15min":4.1840000000000002},"max":{"1min":100.479,"5min":135.98099999999999,"15min":254.899},"last":10.781000000000001},{"interface":"front","average":{"1min":16.834,"5min":16.405999999999999,"15min":18.106999999999999},"min":{"1min":5.2670000000000003,"5min":5.2670000000000003,"15min":5.2670000000000003},"max":{"1min":159.16,"5min":159.16,"15min":270.51600000000002},"last":15.69}]},{"osd":1,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":18.103999999999999,"5min":19.869,"15min":20.129999999999999},"min":{"1min":7.3620000000000001,"5min":7.2350000000000003,"15min":5.1120000000000001},"max":{"1min":122.685,"5min":149.53899999999999,"15min":177.15799999999999},"last":16.943999999999999},{"interface":"front","average":{"1min":16.960999999999999,"5min":18.959,"15min":22.780999999999999},"min":{"1min":6.819,"5min":6.2199999999999998,"15min":5.883},"max":{"1min":119.675,"5min":221.483,"15min":309.04199999999997},"last":12.332000000000001}]},{"osd":2,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":19.047000000000001,"5min":18.303999999999998,"15min":19.811},"min":{"1min":8.1219999999999999,"5min":6.8929999999999998,"15min":5.859},"max":{"1min":145.20400000000001,"5min":145.20400000000001,"15min":231.15600000000001},"last":11.675000000000001},{"interface":"front","average":{"1min":22.48,"5min":21.637,"15min":22.283999999999999},"min":{"1min":13.409000000000001,"5min":11.173999999999999,"15min":7.1890000000000001},"max":{"1min":146.94,"5min":146.94,"15min":197.58699999999999},"last":12.997999999999999}]},{"osd":3,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":17.527000000000001,"5min":249.054,"15min":96.846999999999994},"min":{"1min":5.9400000000000004,"5min":5.5039999999999996,"15min":5.3849999999999998},"max":{"1min":152.352,"5min":7958.2420000000002,"15min":7958.2420000000002},"last":6.9770000000000003},{"interface":"front","average":{"1min":24.044,"5min":252.749,"15min":98.444000000000003},"min":{"1min":8.3620000000000001,"5min":6.7009999999999996,"15min":5.9859999999999998},"max":{"1min":142.91399999999999,"5min":7971.3900000000003,"15min":7971.3900000000003},"last":17.882000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":19.713999999999999,"5min":23.420999999999999,"15min":23.501999999999999},"min":{"1min":9.6539999999999999,"5min":7.5300000000000002,"15min":7.5300000000000002},"max":{"1min":80.417000000000002,"5min":236.994,"15min":236.994},"last":14.23},{"interface":"front","average":{"1min":21.596,"5min":21.725999999999999,"15min":22.199999999999999},"min":{"1min":9.5459999999999994,"5min":8.6479999999999997,"15min":6.7859999999999996},"max":{"1min":154.33000000000001,"5min":228.88200000000001,"15min":228.88200000000001},"last":18.707999999999998}]},{"osd":5,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":26.045999999999999,"5min":21.262,"15min":24.367999999999999},"min":{"1min":6.8289999999999997,"5min":6.5540000000000003,"15min":5.7809999999999997},"max":{"1min":274.59800000000001,"5min":274.59800000000001,"15min":289.22699999999998},"last":7.8209999999999997},{"interface":"front","average":{"1min":26.004000000000001,"5min":24.387,"15min":26.591000000000001},"min":{"1min":12.798,"5min":11.048999999999999,"15min":10.132999999999999},"max":{"1min":148.43799999999999,"5min":173.684,"15min":230.613},"last":14.952}]},{"osd":7,"last update":"Mon Oct 30 17:55:14 2023","interfaces":[{"interface":"back","average":{"1min":19.562000000000001,"5min":22.870999999999999,"15min":24.946999999999999},"min":{"1min":7.1609999999999996,"5min":7.1609999999999996,"15min":7.1609999999999996},"max":{"1min":150.018,"5min":192.72399999999999,"15min":290.22300000000001},"last":9.1470000000000002},{"interface":"front","average":{"1min":23.748000000000001,"5min":25.805,"15min":27.835000000000001},"min":{"1min":10.547000000000001,"5min":9.5440000000000005,"15min":7.4489999999999998},"max":{"1min":158.352,"5min":270.01900000000001,"15min":270.01900000000001},"last":19.661000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542428,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60908,"kb_used_data":7524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94310932,"statfs":{"total":96636764160,"available":96574394368,"internally_reserved":0,"allocated":7704576,"data_stored":4236971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":17.713999999999999,"5min":18.492999999999999,"15min":16.776},"min":{"1min":4.6639999999999997,"5min":3.7290000000000001,"15min":3.7290000000000001},"max":{"1min":84.162999999999997,"5min":278.70299999999997,"15min":278.70299999999997},"last":10.382},{"interface":"front","average":{"1min":15.43,"5min":17.457000000000001,"15min":17.106000000000002},"min":{"1min":5.4299999999999997,"5min":4.9850000000000003,"15min":4.9850000000000003},"max":{"1min":47.259,"5min":174.08099999999999,"15min":190.619},"last":6.8920000000000003}]},{"osd":1,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":20.577999999999999,"5min":18.545000000000002,"15min":20.265999999999998},"min":{"1min":9.3889999999999993,"5min":5.6029999999999998,"15min":5.6029999999999998},"max":{"1min":58.914000000000001,"5min":136.81200000000001,"15min":184.21700000000001},"last":19.858000000000001},{"interface":"front","average":{"1min":17.042999999999999,"5min":17.469000000000001,"15min":20.419},"min":{"1min":6.3929999999999998,"5min":6.1669999999999998,"15min":6.0819999999999999},"max":{"1min":53.759,"5min":139.191,"15min":199.434},"last":18.989000000000001}]},{"osd":2,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":22.640999999999998,"5min":20.824000000000002,"15min":22.341000000000001},"min":{"1min":9.6809999999999992,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":112.958,"5min":279.37099999999998,"15min":279.37099999999998},"last":12.289},{"interface":"front","average":{"1min":26.033000000000001,"5min":22.379000000000001,"15min":20.827999999999999},"min":{"1min":11.018000000000001,"5min":6.9390000000000001,"15min":5.8259999999999996},"max":{"1min":92.144999999999996,"5min":198.02500000000001,"15min":321.54000000000002},"last":8.7750000000000004}]},{"osd":3,"last update":"Mon Oct 30 17:55:16 2023","interfaces":[{"interface":"back","average":{"1min":17.798999999999999,"5min":164.95699999999999,"15min":69.094999999999999},"min":{"1min":7.774,"5min":5.9219999999999997,"15min":5.9219999999999997},"max":{"1min":56.622999999999998,"5min":8219.1820000000007,"15min":8219.1820000000007},"last":13.012},{"interface":"front","average":{"1min":15.321,"5min":158.661,"15min":68.135000000000005},"min":{"1min":7.3440000000000003,"5min":6.3929999999999998,"15min":6.3929999999999998},"max":{"1min":42.534999999999997,"5min":8140.4719999999998,"15min":8140.4719999999998},"last":11.608000000000001}]},{"osd":5,"last update":"Mon Oct 30 17:54:53 2023","interfaces":[{"interface":"back","average":{"1min":18.893000000000001,"5min":22.899000000000001,"15min":23.814},"min":{"1min":9.1790000000000003,"5min":6.4169999999999998,"15min":6.4169999999999998},"max":{"1min":65.632000000000005,"5min":292.22800000000001,"15min":292.22800000000001},"last":13.744999999999999},{"interface":"front","average":{"1min":18.199999999999999,"5min":22.989999999999998,"15min":22.789999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":59.558,"5min":179.67500000000001,"15min":180.28800000000001},"last":14.44}]},{"osd":6,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":31.431000000000001,"5min":27.053999999999998,"15min":24.242999999999999},"min":{"1min":9.5459999999999994,"5min":9.5459999999999994,"15min":6.5810000000000004},"max":{"1min":126.249,"5min":191.995,"15min":191.995},"last":16.02},{"interface":"front","average":{"1min":28.806999999999999,"5min":26.370999999999999,"15min":25.456},"min":{"1min":10.333,"5min":8.2420000000000009,"15min":7.6779999999999999},"max":{"1min":86.864999999999995,"5min":207.30099999999999,"15min":207.30099999999999},"last":9.6760000000000002}]},{"osd":7,"last update":"Mon Oct 30 17:55:08 2023","interfaces":[{"interface":"back","average":{"1min":16.986000000000001,"5min":24.992999999999999,"15min":25.995000000000001},"min":{"1min":8.6899999999999995,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":58.936999999999998,"5min":396.928,"15min":396.928},"last":15.279},{"interface":"front","average":{"1min":21.718,"5min":27.449000000000002,"15min":28.113},"min":{"1min":10.461,"5min":10.461,"15min":7.0860000000000003},"max":{"1min":55.451000000000001,"5min":366.32499999999999,"15min":366.32499999999999},"last":18.006}]}]},{"osd":5,"up_from":13,"seq":55834575132,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79348,"kb_used_data":7916,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292492,"statfs":{"total":96636764160,"available":96555511808,"internally_reserved":0,"allocated":8105984,"data_stored":4634811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":22.184999999999999,"5min":16.451000000000001,"15min":16.408999999999999},"min":{"1min":4.8639999999999999,"5min":3.9910000000000001,"15min":3.9569999999999999},"max":{"1min":154.71299999999999,"5min":154.71299999999999,"15min":155.58799999999999},"last":15.454000000000001},{"interface":"front","average":{"1min":23.321999999999999,"5min":18.834,"15min":17.321000000000002},"min":{"1min":4.6429999999999998,"5min":4.6429999999999998,"15min":4.6429999999999998},"max":{"1min":162.02000000000001,"5min":162.02000000000001,"15min":178.42500000000001},"last":5.0039999999999996}]},{"osd":1,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":30.655000000000001,"5min":18.181999999999999,"15min":20.413},"min":{"1min":5.9710000000000001,"5min":5.0970000000000004,"15min":4.8810000000000002},"max":{"1min":235.82900000000001,"5min":235.82900000000001,"15min":235.82900000000001},"last":8.8900000000000006},{"interface":"front","average":{"1min":23.542999999999999,"5min":16.102,"15min":17.175000000000001},"min":{"1min":5.6929999999999996,"5min":5.6929999999999996,"15min":5.1120000000000001},"max":{"1min":107.85899999999999,"5min":128.197,"15min":278.41300000000001},"last":14.289999999999999}]},{"osd":2,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":29.405000000000001,"5min":19.856999999999999,"15min":19.73},"min":{"1min":10.742000000000001,"5min":7.101,"15min":5.8949999999999996},"max":{"1min":232.47800000000001,"5min":232.47800000000001,"15min":234.09299999999999},"last":9.2590000000000003},{"interface":"front","average":{"1min":21.835999999999999,"5min":17.140999999999998,"15min":19.927},"min":{"1min":7.3819999999999997,"5min":6.5510000000000002,"15min":6.5060000000000002},"max":{"1min":109.47799999999999,"5min":109.47799999999999,"15min":277.52300000000002},"last":15.83}]},{"osd":3,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":34.145000000000003,"5min":154.58500000000001,"15min":65.951999999999998},"min":{"1min":6.3410000000000002,"5min":6.2080000000000002,"15min":6.2080000000000002},"max":{"1min":265.51400000000001,"5min":6785.9989999999998,"15min":6785.9989999999998},"last":16.123999999999999},{"interface":"front","average":{"1min":36.165999999999997,"5min":155.98099999999999,"15min":66.064999999999998},"min":{"1min":9.6110000000000007,"5min":6.6870000000000003,"15min":6.2839999999999998},"max":{"1min":288.66199999999998,"5min":6786.7420000000002,"15min":6786.7420000000002},"last":18.039000000000001}]},{"osd":4,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":29.448,"5min":20.707000000000001,"15min":21.905999999999999},"min":{"1min":8.2300000000000004,"5min":6.9790000000000001,"15min":6.6609999999999996},"max":{"1min":122.157,"5min":143.75800000000001,"15min":244.25},"last":13.318},{"interface":"front","average":{"1min":31.305,"5min":25.204999999999998,"15min":25.847999999999999},"min":{"1min":11.073,"5min":11.073,"15min":8.609},"max":{"1min":106.134,"5min":144.934,"15min":257.089},"last":15.051}]},{"osd":6,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":25.896999999999998,"5min":18.837,"15min":21.152000000000001},"min":{"1min":6.3680000000000003,"5min":6.3680000000000003,"15min":6.3680000000000003},"max":{"1min":169.02199999999999,"5min":169.02199999999999,"15min":280.36000000000001},"last":11.722},{"interface":"front","average":{"1min":27.088999999999999,"5min":22.541,"15min":25.475000000000001},"min":{"1min":9.0850000000000009,"5min":7.4020000000000001,"15min":7.4020000000000001},"max":{"1min":113.434,"5min":113.434,"15min":346.47000000000003},"last":19.763000000000002}]},{"osd":7,"last update":"Mon Oct 30 17:55:09 2023","interfaces":[{"interface":"back","average":{"1min":35.899999999999999,"5min":23.495999999999999,"15min":27.568999999999999},"min":{"1min":7.7530000000000001,"5min":7.7530000000000001,"15min":7.6520000000000001},"max":{"1min":170.19399999999999,"5min":170.19399999999999,"15min":264.64499999999998},"last":20.542000000000002},{"interface":"front","average":{"1min":28.663,"5min":23.901,"15min":27.988},"min":{"1min":10.331,"5min":10.331,"15min":9.9979999999999993},"max":{"1min":96.566999999999993,"5min":145.00899999999999,"15min":282.32400000000001},"last":13.991}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T17:55:52.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T17:55:52.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.632+0000 7f65ed4d7640 10 monclient: get_monmap_and_config 2023-10-30T17:55:52.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.632+0000 7f65ed4d7640 10 monclient: build_initial_monmap 2023-10-30T17:55:52.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.632+0000 7f65ed4d7640 10 monclient: monmap: 2023-10-30T17:55:52.634 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:52.635 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:52.635 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:55:52.633404+0000 2023-10-30T17:55:52.635 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:55:52.633404+0000 2023-10-30T17:55:52.635 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:52.635 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:0: v1:172.21.15.170:6789/0 mon.noname-a-legacy 2023-10-30T17:55:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2023-10-30T17:55:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-b 2023-10-30T17:55:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:3: v2:172.21.15.170:3300/0 mon.noname-a 2023-10-30T17:55:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2023-10-30T17:55:52.637 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 1 Processor -- start 2023-10-30T17:55:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 1 -- start start 2023-10-30T17:55:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient: init 2023-10-30T17:55:52.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:52.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T17:55:52.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f65e80b00c0 0x7f65e8070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient(hunting): picked mon.noname-c con 0x7f65e80b00c0 addr v2:172.21.15.177:3300/0 2023-10-30T17:55:52.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f65e80b04a0 0x7f65e80acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient(hunting): picked mon.noname-a con 0x7f65e80b04a0 addr v2:172.21.15.170:3300/0 2023-10-30T17:55:52.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65e813cd90 0x7f65e8142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient(hunting): picked mon.noname-b con 0x7f65e813cd90 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:52.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e8071020 con 0x7f65e80b04a0 2023-10-30T17:55:52.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e80ad5c0 con 0x7f65e80b00c0 2023-10-30T17:55:52.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80ad310 con 0x7f65e813cd90 2023-10-30T17:55:52.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:52.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.633+0000 7f65ed4d7640 10 monclient(hunting): authenticate will time out at 2734.956818s 2023-10-30T17:55:52.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e67fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f65e80b04a0 0x7f65e80acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65e813cd90 0x7f65e8142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e6ffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f65e80b00c0 0x7f65e8070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e67fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f65e80b04a0 0x7f65e80acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:45560/0 (socket says 172.21.15.170:45560) 2023-10-30T17:55:52.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e67fc640 1 -- 172.21.15.170:0/3515885760 learned_addr learned my addr 172.21.15.170:0/3515885760 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:52.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e67fc640 10 monclient(hunting): get_auth_request con 0x7f65e80b04a0 auth_method 0 2023-10-30T17:55:52.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e77fe640 10 monclient(hunting): get_auth_request con 0x7f65e813cd90 auth_method 0 2023-10-30T17:55:52.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e6ffd640 10 monclient(hunting): get_auth_request con 0x7f65e80b00c0 auth_method 0 2023-10-30T17:55:52.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.634+0000 7f65e6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.635+0000 7f65e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.635+0000 7f65e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.635+0000 7f65e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.635+0000 7f65e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.635+0000 7f65e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.635+0000 7f65e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.635+0000 7f65e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.635+0000 7f65e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.635+0000 7f65e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e67fc640 10 monclient(hunting): handle_auth_done global_id 16098 payload 274 2023-10-30T17:55:52.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e67fc640 1 -- 172.21.15.170:0/3515885760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65e813cd90 msgr2=0x7f65e8142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:52.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e67fc640 1 --2- 172.21.15.170:0/3515885760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65e813cd90 0x7f65e8142cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e67fc640 1 -- 172.21.15.170:0/3515885760 >> v2:172.21.15.177:3300/0 conn(0x7f65e80b00c0 msgr2=0x7f65e8070ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T17:55:52.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e67fc640 1 --2- 172.21.15.170:0/3515885760 >> v2:172.21.15.177:3300/0 conn(0x7f65e80b00c0 0x7f65e8070ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e67fc640 10 monclient: _finish_hunting 0 2023-10-30T17:55:52.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e67fc640 1 monclient: found mon.noname-a 2023-10-30T17:55:52.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:52.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:52.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e67fc640 1 -- 172.21.15.170:0/3515885760 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e8072100 con 0x7f65e80b04a0 2023-10-30T17:55:52.650 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.637+0000 7f65e6ffd640 1 --2- 172.21.15.170:0/3515885760 >> v2:172.21.15.177:3300/0 conn(0x7f65e80b00c0 0x7f65e8070ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T17:55:52.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.639+0000 7f65e67fc640 1 --2- 172.21.15.170:0/3515885760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e80b04a0 0x7f65e80acdd0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7f65e80b3ef0 tx=0x7f65d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514aa3acb155ef58 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:52.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 1 -- 172.21.15.170:0/3515885760 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8031870 con 0x7f65e80b04a0 2023-10-30T17:55:52.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:52.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:52.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 10 monclient: dump: 2023-10-30T17:55:52.655 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:52.655 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:52.655 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.656 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.656 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:52.656 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.656 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:52.656 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:52.657 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:52.657 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 10 monclient: _finish_auth 0 2023-10-30T17:55:52.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 10 monclient: _check_auth_tickets 2023-10-30T17:55:52.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:52.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 1 -- 172.21.15.170:0/3515885760 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d80319d0 con 0x7f65e80b04a0 2023-10-30T17:55:52.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 5 monclient: authenticate success, global_id 16098 2023-10-30T17:55:52.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T17:55:52.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:52.658 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 10 monclient: get_monmap_and_config success 2023-10-30T17:55:52.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 4 set_mon_vals no callback set 2023-10-30T17:55:52.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 1 -- 172.21.15.170:0/3515885760 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8031d60 con 0x7f65e80b04a0 2023-10-30T17:55:52.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:52.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:52.659 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65e5ffb640 10 monclient: dump: 2023-10-30T17:55:52.660 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:52.660 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:52.660 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.660 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.660 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:52.661 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.661 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:52.661 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:52.661 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:52.661 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 10 monclient: shutdown 2023-10-30T17:55:52.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:52.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 1 -- 172.21.15.170:0/3515885760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e80b04a0 msgr2=0x7f65e80acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:52.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 1 --2- 172.21.15.170:0/3515885760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e80b04a0 0x7f65e80acdd0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7f65e80b3ef0 tx=0x7f65d802b910 comp rx=0 tx=0).stop 2023-10-30T17:55:52.662 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 1 -- 172.21.15.170:0/3515885760 shutdown_connections 2023-10-30T17:55:52.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 1 --2- 172.21.15.170:0/3515885760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65e813cd90 0x7f65e8142cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 1 --2- 172.21.15.170:0/3515885760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e80b04a0 0x7f65e80acdd0 unknown :-1 s=CLOSED pgs=3389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 1 --2- 172.21.15.170:0/3515885760 >> v2:172.21.15.177:3300/0 conn(0x7f65e80b00c0 0x7f65e8070ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.643+0000 7f65ed4d7640 1 -- 172.21.15.170:0/3515885760 >> 172.21.15.170:0/3515885760 conn(0x7f65e80b3560 msgr2=0x7f65e80af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:52.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.644+0000 7f65ed4d7640 1 -- 172.21.15.170:0/3515885760 shutdown_connections 2023-10-30T17:55:52.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.644+0000 7f65ed4d7640 1 -- 172.21.15.170:0/3515885760 wait complete. 2023-10-30T17:55:52.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.644+0000 7f65ed4d7640 10 monclient: build_initial_monmap 2023-10-30T17:55:52.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.644+0000 7f65ed4d7640 10 monclient: monmap: 2023-10-30T17:55:52.664 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 0 2023-10-30T17:55:52.664 INFO:teuthology.orchestra.run.smithi170.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T17:55:52.664 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 0.000000 2023-10-30T17:55:52.665 INFO:teuthology.orchestra.run.smithi170.stderr:created 0.000000 2023-10-30T17:55:52.665 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 0 (unknown) 2023-10-30T17:55:52.665 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.665 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.noname-a 2023-10-30T17:55:52.665 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2023-10-30T17:55:52.666 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.noname-c 2023-10-30T17:55:52.666 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.644+0000 7f65ed4d7640 1 librados: starting msgr at 2023-10-30T17:55:52.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.644+0000 7f65ed4d7640 1 librados: starting objecter 2023-10-30T17:55:52.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 1 Processor -- start 2023-10-30T17:55:52.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 1 -- start start 2023-10-30T17:55:52.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 1 librados: setting wanted keys 2023-10-30T17:55:52.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 1 librados: calling monclient init 2023-10-30T17:55:52.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient: init 2023-10-30T17:55:52.667 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient: _reopen_session rank -1 2023-10-30T17:55:52.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T17:55:52.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65e80b00c0 0x7f65e80a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient(hunting): picked mon.noname-b con 0x7f65e80b00c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2023-10-30T17:55:52.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65e813cd90 0x7f65e8070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.668 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient(hunting): picked mon.noname-c con 0x7f65e813cd90 addr [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] 2023-10-30T17:55:52.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e813fbf0 0x7f65e8138790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient(hunting): picked mon.noname-a con 0x7f65e813fbf0 addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2023-10-30T17:55:52.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80a95f0 con 0x7f65e813fbf0 2023-10-30T17:55:52.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80a9210 con 0x7f65e80b00c0 2023-10-30T17:55:52.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient(hunting): start opening mon connection 2023-10-30T17:55:52.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80a9380 con 0x7f65e813cd90 2023-10-30T17:55:52.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient(hunting): _renew_subs 2023-10-30T17:55:52.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.645+0000 7f65ed4d7640 10 monclient(hunting): authenticate will time out at 2734.968878s 2023-10-30T17:55:52.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e813fbf0 0x7f65e8138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e67fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65e813cd90 0x7f65e8070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e813fbf0 0x7f65e8138790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:45564/0 (socket says 172.21.15.170:45564) 2023-10-30T17:55:52.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e77fe640 1 -- 172.21.15.170:0/2569447035 learned_addr learned my addr 172.21.15.170:0/2569447035 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-30T17:55:52.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e77fe640 10 monclient(hunting): get_auth_request con 0x7f65e813fbf0 auth_method 0 2023-10-30T17:55:52.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e6ffd640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65e80b00c0 0x7f65e80a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e67fc640 10 monclient(hunting): get_auth_request con 0x7f65e813cd90 auth_method 0 2023-10-30T17:55:52.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e6ffd640 10 monclient(hunting): get_auth_request con 0x7f65e80b00c0 auth_method 0 2023-10-30T17:55:52.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T17:55:52.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T17:55:52.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.646+0000 7f65e6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T17:55:52.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.647+0000 7f65e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.647+0000 7f65e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.647+0000 7f65e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.647+0000 7f65e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.647+0000 7f65e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.647+0000 7f65e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.647+0000 7f65e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T17:55:52.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.647+0000 7f65e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T17:55:52.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.647+0000 7f65e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T17:55:52.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.649+0000 7f65e77fe640 10 monclient(hunting): handle_auth_done global_id 16101 payload 824 2023-10-30T17:55:52.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.649+0000 7f65e77fe640 1 -- 172.21.15.170:0/2569447035 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65e813cd90 msgr2=0x7f65e8070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:52.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.649+0000 7f65e77fe640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65e813cd90 0x7f65e8070170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.649+0000 7f65e77fe640 1 -- 172.21.15.170:0/2569447035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65e80b00c0 msgr2=0x7f65e80a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:52.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.649+0000 7f65e77fe640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65e80b00c0 0x7f65e80a9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:52.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.649+0000 7f65e77fe640 10 monclient: _finish_hunting 0 2023-10-30T17:55:52.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.649+0000 7f65e77fe640 1 monclient: found mon.noname-a 2023-10-30T17:55:52.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.649+0000 7f65e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T17:55:52.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.649+0000 7f65e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.170:3300/0 2023-10-30T17:55:52.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.649+0000 7f65e77fe640 1 -- 172.21.15.170:0/2569447035 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc0093a0 con 0x7f65e813fbf0 2023-10-30T17:55:52.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.651+0000 7f65e77fe640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e813fbf0 0x7f65e8138790 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009d30 tx=0x7f65e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:52.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.652+0000 7f65cf7fe640 10 client.?.objecter ms_handle_connect 0x7f65e813fbf0 2023-10-30T17:55:52.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.652+0000 7f65cf7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T17:55:52.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 1 -- 172.21.15.170:0/2569447035 <== mon.0 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0009ba0 con 0x7f65e813fbf0 2023-10-30T17:55:52.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:52.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T17:55:52.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 10 monclient: dump: 2023-10-30T17:55:52.680 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:52.680 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:52.680 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.680 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.681 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:52.681 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.681 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:52.681 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:52.681 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:52.682 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 10 monclient: _finish_auth 0 2023-10-30T17:55:52.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 10 monclient: _check_auth_tickets 2023-10-30T17:55:52.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T17:55:52.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 1 -- 172.21.15.170:0/2569447035 <== mon.0 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e000f040 con 0x7f65e813fbf0 2023-10-30T17:55:52.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T17:55:52.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 1 -- 172.21.15.170:0/2569447035 <== mon.0 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e00146c0 con 0x7f65e813fbf0 2023-10-30T17:55:52.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65ed4d7640 5 monclient: authenticate success, global_id 16101 2023-10-30T17:55:52.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65ed4d7640 10 monclient: _renew_subs 2023-10-30T17:55:52.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65ed4d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:52.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65ed4d7640 1 -- 172.21.15.170:0/2569447035 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d80094d0 con 0x7f65e813fbf0 2023-10-30T17:55:52.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T17:55:52.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T17:55:52.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65cf7fe640 10 monclient: dump: 2023-10-30T17:55:52.684 INFO:teuthology.orchestra.run.smithi170.stderr:epoch 1 2023-10-30T17:55:52.684 INFO:teuthology.orchestra.run.smithi170.stderr:fsid dbedaa2b-1947-40d2-95bd-9b4fc7298a39 2023-10-30T17:55:52.685 INFO:teuthology.orchestra.run.smithi170.stderr:last_changed 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.685 INFO:teuthology.orchestra.run.smithi170.stderr:created 2023-10-30T17:30:45.007158+0000 2023-10-30T17:55:52.685 INFO:teuthology.orchestra.run.smithi170.stderr:min_mon_release 18 (reef) 2023-10-30T17:55:52.685 INFO:teuthology.orchestra.run.smithi170.stderr:election_strategy: 1 2023-10-30T17:55:52.685 INFO:teuthology.orchestra.run.smithi170.stderr:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2023-10-30T17:55:52.686 INFO:teuthology.orchestra.run.smithi170.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2023-10-30T17:55:52.686 INFO:teuthology.orchestra.run.smithi170.stderr:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-30T17:55:52.686 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T17:55:52.686 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65e57fa640 4 set_mon_vals no callback set 2023-10-30T17:55:52.686 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65ed4d7640 10 client.16101.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T17:55:52.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65ed4d7640 10 monclient: _renew_subs 2023-10-30T17:55:52.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65ed4d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:52.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65ed4d7640 1 -- 172.21.15.170:0/2569447035 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e8139490 con 0x7f65e813fbf0 2023-10-30T17:55:52.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.655+0000 7f65ed4d7640 1 librados: init done 2023-10-30T17:55:52.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.656+0000 7f65ed4d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:52.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.656+0000 7f65ed4d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:52.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.656+0000 7f65ed4d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:52.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.656+0000 7f65ed4d7640 1 -- 172.21.15.170:0/2569447035 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e804e3c0 con 0x7f65e813fbf0 2023-10-30T17:55:52.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.891+0000 7f65cf7fe640 1 -- 172.21.15.170:0/2569447035 <== mon.0 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f65e0014820 con 0x7f65e813fbf0 2023-10-30T17:55:52.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.892+0000 7f65cf7fe640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f65d406c880 0x7f65d406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T17:55:52.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.892+0000 7f65e6ffd640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f65d406c880 0x7f65d406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T17:55:52.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.892+0000 7f65e6ffd640 10 monclient: get_auth_request con 0x7f65d406c880 auth_method 0 2023-10-30T17:55:52.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.893+0000 7f65e6ffd640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f65d406c880 0x7f65d406ed40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009cc0 tx=0x7f65dc009110 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T17:55:52.895 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.893+0000 7f65cf7fe640 10 client.16101.objecter ms_handle_connect 0x7f65d406c880 2023-10-30T17:55:52.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.909+0000 7f65cf7fe640 1 -- 172.21.15.170:0/2569447035 <== mon.0 v2:172.21.15.170:3300/0 5 ==== osd_map(914..914 src has 737..914) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f65e0059460 con 0x7f65e813fbf0 2023-10-30T17:55:52.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.910+0000 7f65cf7fe640 10 client.16101.objecter ms_dispatch 0x7f65e8000b90 osd_map(914..914 src has 737..914) v4 2023-10-30T17:55:52.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.910+0000 7f65cf7fe640 3 client.16101.objecter handle_osd_map got epochs [914,914] > 0 2023-10-30T17:55:52.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.910+0000 7f65cf7fe640 3 client.16101.objecter handle_osd_map decoding full epoch 914 2023-10-30T17:55:52.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.910+0000 7f65cf7fe640 20 client.16101.objecter dump_active .. 0 homeless 2023-10-30T17:55:52.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.910+0000 7f65cf7fe640 1 -- 172.21.15.170:0/2569447035 <== mon.0 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65e00557a0 con 0x7f65e813fbf0 2023-10-30T17:55:52.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.910+0000 7f65cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T17:55:52.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:52.910+0000 7f65cf7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T17:55:53.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.089+0000 7f65ed4d7640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T17:55:53.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.089+0000 7f65ed4d7640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T17:55:53.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.089+0000 7f65ed4d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.170:3300/0 2023-10-30T17:55:53.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.089+0000 7f65ed4d7640 1 -- 172.21.15.170:0/2569447035 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f65e8139490 con 0x7f65e813fbf0 2023-10-30T17:55:53.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.195+0000 7f65cf7fe640 1 -- 172.21.15.170:0/2569447035 <== mon.0 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f65e8139490 con 0x7f65e813fbf0 2023-10-30T17:55:53.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.195+0000 7f65cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T17:55:53.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.195+0000 7f65cf7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T17:55:53.197 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.195+0000 7f65cf7fe640 1 -- 172.21.15.170:0/2569447035 <== mon.0 v2:172.21.15.170:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f65e005d500 con 0x7f65e813fbf0 2023-10-30T17:55:53.197 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.195+0000 7f65cf7fe640 10 monclient: handle_config config(1 keys) v1 2023-10-30T17:55:53.197 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.195+0000 7f65e57fa640 4 set_mon_vals no callback set 2023-10-30T17:55:53.197 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.195+0000 7f65e57fa640 10 set_mon_vals mon_health_to_clog = false 2023-10-30T17:55:53.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.196+0000 7f65ed4d7640 10 librados: watch_flush enter 2023-10-30T17:55:53.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.196+0000 7f65ed4d7640 10 librados: watch_flush exit 2023-10-30T17:55:53.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.196+0000 7f65ed4d7640 20 client.16101.objecter shutdown clearing up homeless session... 2023-10-30T17:55:53.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.196+0000 7f65ed4d7640 10 client.16101.objecter successfully canceled tick 2023-10-30T17:55:53.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 -- 172.21.15.170:0/2569447035 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f65d406c880 msgr2=0x7f65d406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:53.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f65d406c880 0x7f65d406ed40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009cc0 tx=0x7f65dc009110 comp rx=0 tx=0).stop 2023-10-30T17:55:53.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 10 monclient: shutdown 2023-10-30T17:55:53.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T17:55:53.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 -- 172.21.15.170:0/2569447035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e813fbf0 msgr2=0x7f65e8138790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T17:55:53.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e813fbf0 0x7f65e8138790 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009d30 tx=0x7f65e000c490 comp rx=0 tx=0).stop 2023-10-30T17:55:53.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 -- 172.21.15.170:0/2569447035 shutdown_connections 2023-10-30T17:55:53.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.177:6800/1894191478,v1:172.21.15.177:6801/1894191478] conn(0x7f65d406c880 0x7f65d406ed40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:53.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65e813fbf0 0x7f65e8138790 unknown :-1 s=CLOSED pgs=3390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:53.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65e813cd90 0x7f65e8070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:53.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 --2- 172.21.15.170:0/2569447035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65e80b00c0 0x7f65e80a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T17:55:53.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 -- 172.21.15.170:0/2569447035 >> 172.21.15.170:0/2569447035 conn(0x7f65e80b3560 msgr2=0x7f65e80b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T17:55:53.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 -- 172.21.15.170:0/2569447035 shutdown_connections 2023-10-30T17:55:53.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 -- 172.21.15.170:0/2569447035 wait complete. 2023-10-30T17:55:53.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-30T17:55:53.197+0000 7f65ed4d7640 1 librados: shutdown 2023-10-30T17:55:53.212 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T17:55:53.213 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T17:55:53.213 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T17:55:53.214 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T17:55:53.226 INFO:tasks.ceph.osd.0.smithi170.stderr:2023-10-30T17:55:53.216+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86358) UID: 0 2023-10-30T17:55:53.226 INFO:tasks.ceph.osd.0.smithi170.stderr:2023-10-30T17:55:53.217+0000 b47e640 -1 osd.0 914 *** Got signal Terminated *** 2023-10-30T17:55:57.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-30T17:55:59.315 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T17:55:59.316 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T17:55:59.316 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T17:55:59.331 INFO:tasks.ceph.osd.1.smithi170.stderr:2023-10-30T17:55:59.319+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86367) UID: 0 2023-10-30T17:55:59.331 INFO:tasks.ceph.osd.1.smithi170.stderr:2023-10-30T17:55:59.320+0000 b47e640 -1 osd.1 917 *** Got signal Terminated *** 2023-10-30T17:55:59.715 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:43.681 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:59.718 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:43.684 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:59.741 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:43.707 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:59.777 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:43.743 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:59.779 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:43.718 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:59.781 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:43.747 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:59.785 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:43.724 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:59.799 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:43.765 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:55:59.807 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:43.746 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:03.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-30T17:56:09.700 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:53.666 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:09.709 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:53.675 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:09.734 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:53.700 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:09.758 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:53.697 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:09.760 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:53.700 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:09.762 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:53.728 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:09.767 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:53.733 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:09.776 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:53.715 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:09.785 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:53.751 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:10.346 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:54.285 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:10.347 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:54.286 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:10.355 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:54.294 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:10.419 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:54.385 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:10.420 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:54.386 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:10.429 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:24:54.395 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:10.442 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:54.408 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:10.443 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:54.409 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:10.452 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:24:54.418 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:19.682 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:03.648 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:19.729 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:03.694 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:19.730 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:03.669 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:19.733 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:03.672 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:19.743 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:03.709 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:19.745 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:03.685 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:19.775 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:03.741 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:19.781 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:03.747 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:19.792 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:03.758 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:29.756 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:13.722 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:29.760 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:13.726 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:29.772 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:13.738 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:29.807 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:13.773 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:29.809 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:13.748 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:29.812 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:13.751 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:29.813 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:13.780 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:29.824 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:13.764 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:29.826 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:13.793 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:34.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-30T17:56:35.926 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T17:56:35.927 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T17:56:35.927 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T17:56:35.939 INFO:tasks.ceph.osd.2.smithi170.stderr:2023-10-30T17:56:35.930+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86375) UID: 0 2023-10-30T17:56:35.940 INFO:tasks.ceph.osd.2.smithi170.stderr:2023-10-30T17:56:35.931+0000 b47e640 -1 osd.2 920 *** Got signal Terminated *** 2023-10-30T17:56:39.870 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:23.836 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:39.872 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:23.838 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:39.887 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:23.853 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:39.916 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:23.855 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:39.917 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:23.857 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:39.921 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:23.887 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:39.924 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:23.890 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:39.931 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:23.871 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:39.947 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:23.912 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:40.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-30T17:56:40.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-30T17:56:42.029 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T17:56:42.030 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T17:56:42.030 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T17:56:42.045 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-30T17:56:42.033+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86382) UID: 0 2023-10-30T17:56:42.045 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-30T17:56:42.034+0000 b47e640 -1 osd.3 924 *** Got signal Terminated *** 2023-10-30T17:56:46.104 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-10-30T17:56:46.104 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-30T17:56:48.132 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T17:56:48.133 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T17:56:48.133 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T17:56:48.148 INFO:tasks.ceph.osd.4.smithi177.stderr:2023-10-30T17:56:48.136+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86223) UID: 0 2023-10-30T17:56:48.149 INFO:tasks.ceph.osd.4.smithi177.stderr:2023-10-30T17:56:48.137+0000 b47e640 -1 osd.4 928 *** Got signal Terminated *** 2023-10-30T17:56:49.905 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:33.871 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:49.953 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:33.919 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:49.969 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:33.935 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:49.999 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:33.939 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:50.000 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:33.940 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:50.003 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:33.970 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:50.005 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:33.971 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:50.019 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:33.958 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:50.028 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:33.994 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:52.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-10-30T17:56:54.963 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:38.929 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:54.966 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:38.932 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:54.981 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:38.947 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:55.014 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:38.980 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:55.015 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:38.954 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:55.015 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:38.981 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:55.016 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:38.955 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:55.030 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:38.996 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:56:55.031 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:38.971 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:09.915 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:53.880 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:09.916 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:53.883 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:09.932 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:53.898 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:09.963 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:53.929 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:09.966 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:53.905 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:09.967 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:53.906 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:09.967 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:53.933 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:09.981 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:53.920 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:09.986 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:53.952 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:14.908 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:58.874 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:14.913 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:58.879 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:14.925 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:25:58.891 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:14.956 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:58.922 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:14.958 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:58.897 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:14.961 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:58.927 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:14.961 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:58.900 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:14.971 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:58.911 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:14.974 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:25:58.940 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:29.949 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:13.915 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:29.953 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:13.919 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:29.968 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:13.934 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:29.997 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:13.936 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:30.001 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:13.940 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:30.014 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:13.980 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:30.015 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:13.954 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:30.022 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:13.988 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:30.038 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:14.004 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:34.961 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:18.927 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:34.965 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:18.932 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:34.978 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:18.944 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:35.013 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:18.978 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:35.013 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:18.953 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:35.016 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:18.982 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:35.016 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:18.955 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:35.026 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:18.965 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:35.029 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:18.995 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:49.148 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T17:57:49.149 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T17:57:49.149 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T17:57:49.163 INFO:tasks.ceph.osd.5.smithi177.stderr:2023-10-30T17:57:49.152+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86227) UID: 0 2023-10-30T17:57:49.165 INFO:tasks.ceph.osd.5.smithi177.stderr:2023-10-30T17:57:49.154+0000 b47e640 -1 osd.5 932 *** Got signal Terminated *** 2023-10-30T17:57:50.090 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:34.056 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:50.092 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:34.058 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:50.107 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:34.073 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:50.138 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:34.077 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:50.138 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:34.104 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:50.139 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:34.078 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:50.144 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:34.110 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:50.154 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:34.093 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:50.159 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:34.125 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:55.020 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:38.986 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:55.025 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:38.991 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:55.037 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:39.003 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:55.070 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:39.036 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:55.074 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:39.013 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:55.076 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:39.016 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:55.078 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:39.044 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:55.088 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:39.027 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:55.089 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:39.055 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:57:55.251 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T17:57:55.252 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-30T17:57:55.252 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T17:57:55.264 INFO:tasks.ceph.osd.6.smithi177.stderr:2023-10-30T17:57:55.255+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86229) UID: 0 2023-10-30T17:57:55.264 INFO:tasks.ceph.osd.6.smithi177.stderr:2023-10-30T17:57:55.256+0000 b47e640 -1 osd.6 934 *** Got signal Terminated *** 2023-10-30T17:58:01.354 INFO:tasks.ceph.osd.6:Stopped 2023-10-30T17:58:01.355 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-30T17:58:01.355 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T17:58:01.369 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:58:01.358+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86231) UID: 0 2023-10-30T17:58:01.373 INFO:tasks.ceph.osd.7.smithi177.stderr:2023-10-30T17:58:01.360+0000 b47e640 -1 osd.7 934 *** Got signal Terminated *** 2023-10-30T17:58:10.073 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:54.039 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:10.074 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:54.040 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:10.088 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:54.054 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:10.120 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:54.059 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:10.121 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:54.087 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:10.122 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:54.061 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:10.122 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:54.088 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:10.142 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:54.081 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:10.143 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:54.108 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:14.666 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:58.632 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:14.674 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:58.640 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:14.688 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:58.654 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:14.721 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:58.687 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:14.722 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:58.662 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:14.723 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:58.689 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:14.724 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:58.663 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:14.738 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:58.703 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:14.738 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:58.677 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:15.215 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:59.181 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:15.218 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:59.184 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:15.277 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:59.216 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:15.280 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:59.219 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:15.281 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:26:59.247 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:15.288 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:59.227 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:15.310 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:59.276 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:15.311 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:59.277 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:15.319 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:26:59.285 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:30.176 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:14.142 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:30.182 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:14.148 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:30.200 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:14.166 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:30.226 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:14.165 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:30.229 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:14.168 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:30.230 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:14.196 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:30.232 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:14.198 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:30.244 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:14.183 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:30.247 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:14.213 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:35.197 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:19.163 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:35.199 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:19.165 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:35.213 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:19.179 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:35.245 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:19.185 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:35.246 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:19.186 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:35.248 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:19.214 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:35.249 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:19.216 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:35.256 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:19.196 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:35.261 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:19.227 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:50.238 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:34.203 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:50.241 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:34.207 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:50.266 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:34.232 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:50.294 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:34.260 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:50.298 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:34.264 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:50.320 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:34.286 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:50.322 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:34.261 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:50.324 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:34.264 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:50.342 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:34.281 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:55.254 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:39.219 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:55.256 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:39.222 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:55.269 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:39.235 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:55.301 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:39.240 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:55.303 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:39.242 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:55.305 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:39.271 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:55.306 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:39.272 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:55.313 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:39.252 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:58:55.317 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:39.283 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:10.293 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:54.258 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:10.296 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:54.235 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:10.297 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:54.236 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:10.299 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:54.265 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:10.320 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:54.259 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:10.320 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:54.286 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:10.350 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:54.316 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:10.351 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:54.317 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:10.371 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:54.337 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:15.313 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:59.279 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:15.315 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:59.281 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:15.325 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:27:59.291 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:15.362 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:59.328 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:15.364 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:59.303 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:15.365 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:59.330 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:15.365 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:59.304 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:15.377 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:27:59.343 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:15.379 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:59.318 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:30.328 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:14.294 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:30.374 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:14.314 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:30.375 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:14.341 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:30.379 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:14.318 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:30.403 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:14.343 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:30.404 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:14.370 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:30.429 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:14.395 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:30.432 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:14.398 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:30.449 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:14.415 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:35.370 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:19.335 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:35.374 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:19.340 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:35.387 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:19.353 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:35.421 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:19.360 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:35.422 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:19.388 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:35.425 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:19.364 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:35.426 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:19.391 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:35.436 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:19.402 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:35.441 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:19.380 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:50.424 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:34.390 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:50.429 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:34.395 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:50.448 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:34.413 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:50.478 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:34.417 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:50.479 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:34.445 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:50.481 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:34.447 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:50.483 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:34.423 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:50.497 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:34.464 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:50.504 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:34.444 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:55.439 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:39.405 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:55.444 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:39.410 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:55.456 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:39.422 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:55.492 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:39.431 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:55.494 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:39.434 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:55.496 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:39.462 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:55.501 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:39.467 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:55.508 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:39.448 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T17:59:55.511 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:39.477 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:10.439 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:54.405 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:10.489 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:54.455 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:10.508 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:54.474 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:10.536 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:54.475 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:10.553 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:54.519 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:10.554 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:54.492 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:10.557 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:54.523 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:10.569 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:54.509 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:10.573 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:54.539 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:15.501 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:59.467 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:15.505 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:59.472 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:15.519 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:28:59.485 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:15.552 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:59.491 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:15.553 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:59.492 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:15.554 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:59.520 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:15.557 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:59.523 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:15.564 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:59.503 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:15.568 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:28:59.534 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:30.545 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:14.510 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:30.549 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:14.516 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:30.574 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:14.540 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:30.594 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:14.533 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:30.597 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:14.537 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:30.600 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:14.566 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:30.605 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:14.571 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:30.613 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:14.552 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:30.623 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:14.589 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:35.556 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:19.521 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:35.560 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:19.526 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:35.573 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:19.539 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:35.606 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:19.572 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:35.606 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:19.545 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:35.609 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:19.548 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:35.609 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:19.576 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:35.619 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:19.558 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:35.620 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:19.586 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:50.644 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:34.610 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:50.646 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:34.612 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:50.665 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:34.631 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:50.693 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:34.632 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:50.696 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:34.636 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:50.698 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:34.664 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:50.700 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:34.666 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:50.711 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:34.651 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:50.716 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:34.682 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:55.655 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:39.621 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:55.658 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:39.625 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:55.671 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:39.637 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:55.704 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:39.644 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:55.706 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:39.672 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:55.707 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:39.646 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:55.708 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:39.675 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:55.720 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:39.686 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:00:55.721 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:39.661 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:10.653 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:54.617 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:10.699 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:54.665 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:10.716 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:54.682 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:10.742 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:54.681 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:10.747 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:54.686 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:10.748 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:54.715 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:10.752 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:54.718 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:10.761 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:54.701 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:10.774 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:54.740 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:15.711 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:59.677 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:15.712 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:59.678 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:15.724 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:29:59.690 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:15.756 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:59.722 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:15.757 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:59.696 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:15.759 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:59.725 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:15.760 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:59.699 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:15.770 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:29:59.736 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:15.771 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:59.710 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:26.061 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:10.027 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:26.066 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:10.032 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:26.084 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:10.050 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:26.116 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:10.056 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:26.121 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:10.060 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:26.133 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:10.099 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:26.138 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:10.104 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:26.140 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:10.079 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:26.155 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:10.121 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:30.755 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:14.720 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:30.757 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:14.723 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:30.769 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:14.735 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:30.800 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:14.766 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:30.805 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:14.744 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:30.805 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:14.771 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:30.808 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:14.747 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:30.816 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:14.782 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:30.817 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:14.757 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:35.767 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:19.732 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:35.770 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:19.736 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:35.782 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:19.748 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:35.815 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:19.754 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:35.816 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:19.782 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:35.817 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:19.757 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:35.818 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:19.784 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:35.829 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:19.795 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:35.830 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:19.769 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:50.804 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:34.770 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:50.808 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:34.774 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:50.833 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:34.799 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:50.857 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:34.796 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:50.859 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:34.799 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:50.863 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:34.829 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:50.865 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:34.832 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:50.876 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:34.815 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:50.883 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:34.849 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:55.894 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:39.860 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:55.935 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:39.901 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:55.940 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:39.879 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:55.944 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:39.883 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:55.948 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:39.914 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:55.957 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:39.923 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:55.960 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:39.926 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:55.981 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:39.920 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:01:55.998 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:39.964 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:10.892 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:54.858 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:10.894 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:54.860 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:10.921 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:54.887 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:10.940 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.879 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:10.943 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.882 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:10.952 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:54.918 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:10.955 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:54.921 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:10.961 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.900 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:10.974 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:54.940 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:15.904 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:59.870 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:15.908 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:59.874 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:15.921 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:30:59.887 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:15.956 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:59.895 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:15.957 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:59.923 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:15.958 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:59.898 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:15.960 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:59.926 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:15.969 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:59.909 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:15.971 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:30:59.937 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:30.953 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:14.919 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:30.955 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:14.921 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:30.973 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:14.939 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:31.002 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:14.941 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:31.004 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:14.943 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:31.008 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:14.974 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:31.009 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:14.976 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:31.020 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:14.959 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:31.027 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:14.993 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:35.962 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:19.927 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:35.964 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:19.930 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:35.976 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:19.942 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:36.011 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:19.950 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:36.012 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:19.978 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:36.013 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:19.952 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:36.014 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:19.980 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:36.024 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:19.963 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:36.025 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:19.991 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:50.959 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:34.925 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:51.007 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:34.946 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:51.009 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:34.975 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:51.011 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:34.950 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:51.027 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:34.966 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:51.028 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:34.994 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:51.059 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:35.026 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:51.062 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:35.028 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:51.078 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:35.044 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:56.016 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:39.981 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:56.021 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:39.987 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:56.022 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:39.961 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:56.025 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:39.965 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:56.038 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:39.978 85690== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:56.039 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:40.006 86074== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:56.071 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:40.037 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:56.074 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:40.040 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:02:56.086 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:40.052 86078== Warning: unimplemented fcntl command: 1036 2023-10-30T18:03:00.424 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T18:03:00.425 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-30T18:03:00.426 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T18:03:00.449 INFO:tasks.ceph.mgr.y:Stopped 2023-10-30T18:03:00.450 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T18:03:00.450 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T18:03:00.451 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T18:03:00.449+0000 7f2b9533a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 85688) UID: 0 2023-10-30T18:03:00.451 INFO:tasks.ceph.mgr.x.smithi177.stderr:2023-10-30T18:03:00.449+0000 7f2b9533a640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-10-30T18:03:00.473 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T18:03:00.474 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T18:03:00.474 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T18:03:00.474 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T18:03:00.504 INFO:tasks.ceph.mon.a.smithi170.stderr:2023-10-30T18:03:00.492+0000 9f61640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86043) UID: 0 2023-10-30T18:03:00.504 INFO:tasks.ceph.mon.a.smithi170.stderr:2023-10-30T18:03:00.494+0000 9f61640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-30T18:03:01.090 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:45.056 86074== 2023-10-30T18:03:01.091 INFO:tasks.ceph.mon.a.smithi170.stderr:==00:00:31:45.056 86074== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T18:03:01.108 INFO:tasks.ceph.mon.a.smithi170.stderr:daemon-helper: command failed with exit status 42 2023-10-30T18:03:04.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-30T18:03:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T18:03:06.577 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-10-30T18:03:06.578 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T18:03:06.579 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T18:03:06.579 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T18:03:06.596 INFO:tasks.ceph.mon.c.smithi170.stderr:2023-10-30T18:03:06.582+0000 9f61640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86045) UID: 0 2023-10-30T18:03:06.596 INFO:tasks.ceph.mon.c.smithi170.stderr:2023-10-30T18:03:06.583+0000 9f61640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-10-30T18:03:07.131 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:51.096 86078== 2023-10-30T18:03:07.131 INFO:tasks.ceph.mon.c.smithi170.stderr:==00:00:31:51.096 86078== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T18:03:07.146 INFO:tasks.ceph.mon.c.smithi170.stderr:daemon-helper: command failed with exit status 42 2023-10-30T18:03:09.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-10-30T18:03:09.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-30T18:03:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T18:03:12.681 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-10-30T18:03:12.682 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T18:03:12.683 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T18:03:12.683 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T18:03:12.697 INFO:tasks.ceph.mon.b.smithi177.stderr:2023-10-30T18:03:12.686+0000 9f61640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 85686) UID: 0 2023-10-30T18:03:12.697 INFO:tasks.ceph.mon.b.smithi177.stderr:2023-10-30T18:03:12.687+0000 9f61640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-10-30T18:03:15.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-10-30T18:03:34.213 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:18.152 85690== 2023-10-30T18:03:34.213 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:18.152 85690== Exit program on first error (--exit-on-first-error=yes) 2023-10-30T18:03:34.228 INFO:tasks.ceph.mon.b.smithi177.stderr:daemon-helper: command failed with exit status 42 2023-10-30T18:03:35.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-10-30T18:03:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-10-30T18:03:37.090 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-10-30T18:03:37.091 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T18:03:37.091 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T18:03:37.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-10-30T18:03:37.195 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T18:03:37.196 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T18:03:37.275 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T18:03:37.275 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T18:03:37.350 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T18:03:37.350 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T18:03:37.414 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T18:03:37.415 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T18:03:37.482 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T18:03:37.482 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T18:03:37.660 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T18:03:37.661 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T18:03:37.742 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T18:03:37.743 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-30T18:03:37.797 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T18:03:37.798 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-30T18:03:37.868 INFO:tasks.ceph:Archiving mon data... 2023-10-30T18:03:37.870 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201/data/mon.a.tgz 2023-10-30T18:03:37.870 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-10-30T18:03:37.887 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.UJz9kFC2ll 2023-10-30T18:03:37.888 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.UJz9kFC2ll 2023-10-30T18:03:38.068 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.UJz9kFC2ll 2023-10-30T18:03:38.154 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.UJz9kFC2ll is 414KB 2023-10-30T18:03:38.254 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.UJz9kFC2ll 2023-10-30T18:03:38.269 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201/data/mon.c.tgz 2023-10-30T18:03:38.270 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-10-30T18:03:38.324 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.rBEueU41n4 2023-10-30T18:03:38.324 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.rBEueU41n4 2023-10-30T18:03:38.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.rBEueU41n4 2023-10-30T18:03:38.547 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.rBEueU41n4 is 415KB 2023-10-30T18:03:38.653 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.rBEueU41n4 2023-10-30T18:03:38.670 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201/data/mon.b.tgz 2023-10-30T18:03:38.672 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2023-10-30T18:03:38.688 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.otumGjYqN6 2023-10-30T18:03:38.689 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.otumGjYqN6 2023-10-30T18:03:38.841 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.otumGjYqN6 2023-10-30T18:03:38.925 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.otumGjYqN6 is 415KB 2023-10-30T18:03:39.020 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.otumGjYqN6 2023-10-30T18:03:39.035 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T18:03:39.036 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T18:03:39.039 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T18:03:39.104 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-30T18:03:39.105 DEBUG:teuthology.orchestra.run.smithi170:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T18:03:39.107 DEBUG:teuthology.orchestra.run.smithi177:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T18:03:39.146 INFO:teuthology.orchestra.run.smithi170.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2023-10-30T18:03:39.146 INFO:teuthology.orchestra.run.smithi170.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2023-10-30T18:03:39.147 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2023-10-30T18:03:39.147 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2023-10-30T18:03:39.148 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2023-10-30T18:03:39.148 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2023-10-30T18:03:39.183 INFO:teuthology.orchestra.run.smithi177.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2023-10-30T18:03:39.184 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2023-10-30T18:03:39.185 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2023-10-30T18:03:39.185 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T18:03:39.187 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201/remote/smithi170/crash 2023-10-30T18:03:39.188 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T18:03:39.215 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201/remote/smithi177/crash 2023-10-30T18:03:39.217 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T18:03:39.248 INFO:tasks.ceph:Compressing logs... 2023-10-30T18:03:39.249 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T18:03:39.260 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T18:03:39.315 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T18:03:39.315 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91745.log 2023-10-30T18:03:39.315 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158107.log 2023-10-30T18:03:39.315 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157687.log 2023-10-30T18:03:39.316 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.158107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89912.log 2023-10-30T18:03:39.316 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157855.log 2023-10-30T18:03:39.316 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157687.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158107.log.gz 2023-10-30T18:03:39.316 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103119.log 2023-10-30T18:03:39.316 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111183.log 2023-10-30T18:03:39.317 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157855.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.157687.log.gz 2023-10-30T18:03:39.317 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.103119.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.89912.log.gz 2023-10-30T18:03:39.317 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131354.log 2023-10-30T18:03:39.317 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.111183.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.157855.log.gz 2023-10-30T18:03:39.317 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92302.log 2023-10-30T18:03:39.318 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.131354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86387.log 2023-10-30T18:03:39.318 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108087.log 2023-10-30T18:03:39.318 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.86387.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-10-30T18:03:39.318 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86387.log.gz 2023-10-30T18:03:39.318 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.108087.log: gzip -5 91.2% -- replaced with /var/log/ceph/ceph-client.admin.103119.log.gz 2023-10-30T18:03:39.319 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-10-30T18:03:39.319 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-10-30T18:03:39.319 INFO:teuthology.orchestra.run.smithi170.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-10-30T18:03:39.319 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/valgrind/osd.2.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.131354.log.gz 2023-10-30T18:03:39.319 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 64.0% --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-10-30T18:03:39.319 INFO:teuthology.orchestra.run.smithi170.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.111183.log.gz 2023-10-30T18:03:39.320 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-10-30T18:03:39.320 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/valgrind/osd.0.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92302.log.gz 2023-10-30T18:03:39.320 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-10-30T18:03:39.320 INFO:teuthology.orchestra.run.smithi170.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-10-30T18:03:39.320 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-10-30T18:03:39.321 INFO:teuthology.orchestra.run.smithi170.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-10-30T18:03:39.321 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91709.log 2023-10-30T18:03:39.321 INFO:teuthology.orchestra.run.smithi170.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-10-30T18:03:39.321 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92229.log 2023-10-30T18:03:39.321 INFO:teuthology.orchestra.run.smithi170.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.108087.log.gz 2023-10-30T18:03:39.322 INFO:teuthology.orchestra.run.smithi170.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-10-30T18:03:39.322 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157477.log 2023-10-30T18:03:39.322 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90105.log 2023-10-30T18:03:39.322 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92229.log: /var/log/ceph/ceph-client.admin.157477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158317.log 2023-10-30T18:03:39.322 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90105.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90712.log 2023-10-30T18:03:39.322 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92229.log.gz 2023-10-30T18:03:39.323 INFO:teuthology.orchestra.run.smithi170.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.157477.log.gz/var/log/ceph/ceph-client.admin.158317.log: 2023-10-30T18:03:39.323 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131353.log 2023-10-30T18:03:39.323 INFO:teuthology.orchestra.run.smithi170.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90105.log.gz 2023-10-30T18:03:39.323 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91810.log 2023-10-30T18:03:39.323 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90712.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92207.log 2023-10-30T18:03:39.324 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.admin.158317.log.gz/var/log/ceph/ceph-client.admin.131353.log: 2023-10-30T18:03:39.324 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90063.log 2023-10-30T18:03:39.324 INFO:teuthology.orchestra.run.smithi170.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90712.log.gz 2023-10-30T18:03:39.324 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92207.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80220.log 2023-10-30T18:03:39.324 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111184.log 2023-10-30T18:03:39.325 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90063.log.gz 2023-10-30T18:03:39.325 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.tmp-client.admin.80220.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80220.log.gz 2023-10-30T18:03:39.325 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.92207.log.gz 2023-10-30T18:03:39.325 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91565.log 2023-10-30T18:03:39.325 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.111184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91591.log 2023-10-30T18:03:39.325 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91883.log 2023-10-30T18:03:39.326 INFO:teuthology.orchestra.run.smithi170.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.91709.log.gz 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111184.log.gz 2023-10-30T18:03:39.326 INFO:teuthology.orchestra.run.smithi170.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.131353.log.gz 2023-10-30T18:03:39.326 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T18:03:39.326 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92301.log 2023-10-30T18:03:39.326 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86085.log 2023-10-30T18:03:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92325.log 2023-10-30T18:03:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.86085.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.91565.log.gz 2023-10-30T18:03:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90913.log 2023-10-30T18:03:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92325.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.86085.log.gz 2023-10-30T18:03:39.327 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92092.log 2023-10-30T18:03:39.328 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90913.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90913.log.gz 2023-10-30T18:03:39.328 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.91810.log.gz 2023-10-30T18:03:39.328 INFO:teuthology.orchestra.run.smithi170.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.92301.log.gz 2023-10-30T18:03:39.328 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158023.log 2023-10-30T18:03:39.328 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157897.log 2023-10-30T18:03:39.328 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.158023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121197.log 2023-10-30T18:03:39.329 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157897.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.92325.log.gz 2023-10-30T18:03:39.329 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158023.log.gz 2023-10-30T18:03:39.329 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108086.log 88.5% -- replaced with /var/log/ceph/ceph-client.admin.157897.log.gz 2023-10-30T18:03:39.329 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T18:03:39.329 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.121197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86414.log 2023-10-30T18:03:39.330 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.108086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158275.log 2023-10-30T18:03:39.330 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.86414.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.108086.log.gz 2023-10-30T18:03:39.330 INFO:teuthology.orchestra.run.smithi170.stderr: 87.0% 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92092.log.gz 2023-10-30T18:03:39.330 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86414.log.gz 2023-10-30T18:03:39.330 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T18:03:39.331 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.158275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92298.log 2023-10-30T18:03:39.331 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90007.log 2023-10-30T18:03:39.331 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158275.log.gz 2023-10-30T18:03:39.331 INFO:teuthology.orchestra.run.smithi170.stderr: 90.9%/var/log/ceph/ceph-client.admin.92298.log: -- replaced with /var/log/ceph/ceph-client.admin.121197.log.gz 2023-10-30T18:03:39.331 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91053.log 2023-10-30T18:03:39.331 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.90007.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.91883.log.gz 2023-10-30T18:03:39.332 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/ceph-client.admin.158233.log 2023-10-30T18:03:39.332 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.92298.log.gz/var/log/ceph/ceph-client.admin.91053.log: 2023-10-30T18:03:39.332 INFO:teuthology.orchestra.run.smithi170.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90007.log.gz 2023-10-30T18:03:39.332 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103120.log 2023-10-30T18:03:39.332 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91053.log.gz 2023-10-30T18:03:39.333 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.158233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91207.log 2023-10-30T18:03:39.333 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5/var/log/ceph/ceph-client.admin.103120.log: --verbose -- /var/log/ceph/ceph-client.admin.91518.log 2023-10-30T18:03:39.333 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91207.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158233.log.gz 2023-10-30T18:03:39.333 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157813.log 2023-10-30T18:03:39.333 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91207.log.gz 2023-10-30T18:03:39.334 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/ceph-client.admin.121196.log 2023-10-30T18:03:39.334 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91518.log: /var/log/ceph/ceph-client.admin.157813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113231.log 2023-10-30T18:03:39.334 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.121196.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.157813.log.gz 2023-10-30T18:03:39.334 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90377.log 2023-10-30T18:03:39.334 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.113231.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.103120.log.gz 2023-10-30T18:03:39.335 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158065.log 2023-10-30T18:03:39.335 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90377.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90377.log.gz 2023-10-30T18:03:39.335 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90373.log 2023-10-30T18:03:39.335 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.158065.log: 90.7% 90.4% -- replaced with /var/log/ceph/ceph-client.admin.121196.log.gz 2023-10-30T18:03:39.335 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.admin.113231.log.gz 2023-10-30T18:03:39.336 INFO:teuthology.orchestra.run.smithi170.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158065.log.gz 2023-10-30T18:03:39.336 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-30T18:03:39.336 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92016.log 2023-10-30T18:03:39.336 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91553.log 2023-10-30T18:03:39.336 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90373.log.gz 2023-10-30T18:03:39.336 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157519.log 2023-10-30T18:03:39.354 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-30T18:03:39.355 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85923.log 2023-10-30T18:03:39.355 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-10-30T18:03:39.355 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.85923.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-10-30T18:03:39.355 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-10-30T18:03:39.356 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.7.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85923.log.gz 2023-10-30T18:03:39.356 INFO:teuthology.orchestra.run.smithi177.stderr: 55.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-10-30T18:03:39.356 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-10-30T18:03:39.356 INFO:teuthology.orchestra.run.smithi177.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-10-30T18:03:39.356 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-10-30T18:03:39.356 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.6.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-10-30T18:03:39.357 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86134.log 2023-10-30T18:03:39.357 INFO:teuthology.orchestra.run.smithi177.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-10-30T18:03:39.357 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T18:03:39.357 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.4.log: /var/log/ceph/ceph-client.admin.86134.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-10-30T18:03:39.357 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86003.log 2023-10-30T18:03:39.358 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T18:03:39.358 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86134.log.gz 2023-10-30T18:03:39.358 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.86003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85843.log 2023-10-30T18:03:39.358 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-30T18:03:39.358 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.85843.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86003.log.gz 2023-10-30T18:03:39.359 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T18:03:39.359 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85883.log 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85843.log.gz 2023-10-30T18:03:39.359 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T18:03:39.359 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T18:03:39.359 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.85883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86043.log 2023-10-30T18:03:39.359 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mgr.x.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85883.log.gz 2023-10-30T18:03:39.360 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86083.log 2023-10-30T18:03:39.360 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.86043.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86043.log.gz 2023-10-30T18:03:39.360 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85963.log 2023-10-30T18:03:39.360 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.86083.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T18:03:39.360 INFO:teuthology.orchestra.run.smithi177.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86083.log.gz/var/log/ceph/ceph-client.admin.85963.log: 2023-10-30T18:03:39.361 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T18:03:39.361 INFO:teuthology.orchestra.run.smithi177.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85963.log.gz 2023-10-30T18:03:39.361 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.log: 92.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T18:03:39.361 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mon.b.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T18:03:39.374 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91553.log: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T18:03:39.374 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90711.log 2023-10-30T18:03:39.375 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157519.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.157519.log.gz 2023-10-30T18:03:39.375 INFO:teuthology.orchestra.run.smithi170.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.92016.log.gz 2023-10-30T18:03:39.375 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.91553.log.gz 2023-10-30T18:03:39.375 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121195.log 2023-10-30T18:03:39.375 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.90711.log: -- /var/log/ceph/ceph-client.admin.90374.log 2023-10-30T18:03:39.376 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.121195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157603.log 2023-10-30T18:03:39.376 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90711.log.gz 2023-10-30T18:03:39.376 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87189.log 2023-10-30T18:03:39.376 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157603.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90374.log.gz 2023-10-30T18:03:39.376 INFO:teuthology.orchestra.run.smithi170.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.157603.log.gz 2023-10-30T18:03:39.377 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92022.log 2023-10-30T18:03:39.377 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91700.log 2023-10-30T18:03:39.377 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92022.log: /var/log/ceph/ceph-client.admin.87189.log: 90.4% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87189.log.gz 2023-10-30T18:03:39.377 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.admin.121195.log.gz 2023-10-30T18:03:39.377 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157435.log 2023-10-30T18:03:39.377 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90953.log 2023-10-30T18:03:39.378 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91700.log: /var/log/ceph/ceph-client.admin.157435.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.157435.log.gz 2023-10-30T18:03:39.378 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91975.log 2023-10-30T18:03:39.378 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90953.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90953.log.gz 2023-10-30T18:03:39.378 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90702.log 2023-10-30T18:03:39.378 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91975.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92022.log.gz 2023-10-30T18:03:39.379 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88194.log 2023-10-30T18:03:39.379 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90702.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90702.log.gz 2023-10-30T18:03:39.379 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92010.log 2023-10-30T18:03:39.379 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.88194.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88194.log.gz 2023-10-30T18:03:39.379 INFO:teuthology.orchestra.run.smithi170.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.91975.log.gz 2023-10-30T18:03:39.379 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92327.log 2023-10-30T18:03:39.380 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91863.log 2023-10-30T18:03:39.380 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92327.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.92327.log.gz 2023-10-30T18:03:39.380 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113229.log 2023-10-30T18:03:39.380 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91863.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.91863.log.gz 2023-10-30T18:03:39.380 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89417.log 2023-10-30T18:03:39.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.113229.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.92010.log.gz 2023-10-30T18:03:39.381 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90376.log 2023-10-30T18:03:39.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89417.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.113229.log.gz 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89417.log.gz 2023-10-30T18:03:39.381 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T18:03:39.381 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T18:03:39.382 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90973.log 2023-10-30T18:03:39.382 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.3.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90376.log.gz 2023-10-30T18:03:39.382 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91495.log 2023-10-30T18:03:39.382 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90973.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90973.log.gz 2023-10-30T18:03:39.382 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90195.log 2023-10-30T18:03:39.382 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91495.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.91591.log.gz 2023-10-30T18:03:39.383 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91123.log 2023-10-30T18:03:39.383 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90195.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90195.log.gz 2023-10-30T18:03:39.383 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158149.log 2023-10-30T18:03:39.383 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91123.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91123.log.gz 2023-10-30T18:03:39.383 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157729.log 2023-10-30T18:03:39.384 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.158149.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158149.log.gz 2023-10-30T18:03:39.384 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90707.log 2023-10-30T18:03:39.384 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157729.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.157729.log.gz 2023-10-30T18:03:39.384 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103118.log 2023-10-30T18:03:39.384 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90707.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90707.log.gz 2023-10-30T18:03:39.384 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91531.log 2023-10-30T18:03:39.385 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.103118.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.103118.log.gz 2023-10-30T18:03:39.385 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121194.log 2023-10-30T18:03:39.385 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91531.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.91518.log.gz 2023-10-30T18:03:39.385 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91050.log 2023-10-30T18:03:39.385 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.121194.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.121194.log.gz 2023-10-30T18:03:39.386 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157645.log 2023-10-30T18:03:39.386 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91050.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91050.log.gz 2023-10-30T18:03:39.386 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-30T18:03:39.386 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157645.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.157645.log.gz 2023-10-30T18:03:39.386 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86512.log 2023-10-30T18:03:39.391 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mgr.y.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.91531.log.gz 2023-10-30T18:03:39.391 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91924.log 2023-10-30T18:03:39.391 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.86512.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86512.log.gz 2023-10-30T18:03:39.391 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92143.log 2023-10-30T18:03:39.392 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91924.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.91924.log.gz 2023-10-30T18:03:39.392 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90375.log 2023-10-30T18:03:39.392 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92143.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.91495.log.gz 2023-10-30T18:03:39.392 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157939.log 2023-10-30T18:03:39.392 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90375.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90375.log.gz 2023-10-30T18:03:39.393 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92183.log 2023-10-30T18:03:39.393 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157939.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.157939.log.gz 2023-10-30T18:03:39.393 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91706.log 2023-10-30T18:03:39.394 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92183.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.92183.log.gz 2023-10-30T18:03:39.397 INFO:teuthology.orchestra.run.smithi170.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-30T18:03:39.398 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90372.log 2023-10-30T18:03:39.398 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121193.log 2023-10-30T18:03:39.399 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90372.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90372.log.gz 2023-10-30T18:03:39.399 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157561.log 2023-10-30T18:03:39.402 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.121193.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.121193.log.gz 2023-10-30T18:03:39.403 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90147.log 2023-10-30T18:03:39.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157561.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157561.log.gz 2023-10-30T18:03:39.404 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T18:03:39.404 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90147.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90147.log.gz 2023-10-30T18:03:39.405 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91604.log 2023-10-30T18:03:39.418 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92059.log 2023-10-30T18:03:39.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91604.log: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T18:03:39.425 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91604.log.gz 2023-10-30T18:03:39.433 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91719.log 2023-10-30T18:03:39.433 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.92143.log.gz 2023-10-30T18:03:39.433 INFO:teuthology.orchestra.run.smithi170.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92059.log.gz 2023-10-30T18:03:39.434 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89951.log 2023-10-30T18:03:39.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91719.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.91719.log.gz 2023-10-30T18:03:39.441 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-30T18:03:39.442 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89951.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89951.log.gz 2023-10-30T18:03:39.443 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113228.log 2023-10-30T18:03:39.456 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.2.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T18:03:39.459 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158191.log 2023-10-30T18:03:39.462 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.113228.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.113228.log.gz 2023-10-30T18:03:39.475 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90704.log 2023-10-30T18:03:39.476 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.158191.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158191.log.gz 2023-10-30T18:03:39.494 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90378.log 2023-10-30T18:03:39.495 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90704.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90704.log.gz 2023-10-30T18:03:39.503 INFO:teuthology.orchestra.run.smithi170.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.91706.log.gz 2023-10-30T18:03:39.504 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90717.log 2023-10-30T18:03:39.504 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92285.log 2023-10-30T18:03:39.504 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5%/var/log/ceph/ceph-client.admin.90717.log: -- replaced with /var/log/ceph/ceph-client.admin.90378.log.gz 2023-10-30T18:03:39.504 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90717.log.gz 2023-10-30T18:03:39.505 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158359.log 2023-10-30T18:03:39.509 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92285.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.92285.log.gz 2023-10-30T18:03:39.509 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90716.log 2023-10-30T18:03:39.510 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.158359.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158359.log.gz 2023-10-30T18:03:39.511 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108085.log 2023-10-30T18:03:39.512 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90716.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90716.log.gz 2023-10-30T18:03:39.512 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90381.log 2023-10-30T18:03:39.516 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.108085.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.108085.log.gz 2023-10-30T18:03:39.516 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158399.log 2023-10-30T18:03:39.527 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90381.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90381.log.gz 2023-10-30T18:03:39.527 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157771.log 2023-10-30T18:03:39.528 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.158399.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158399.log.gz 2023-10-30T18:03:39.528 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90709.log 2023-10-30T18:03:39.528 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157771.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.157771.log.gz 2023-10-30T18:03:39.529 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157981.log 2023-10-30T18:03:39.529 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90709.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90709.log.gz 2023-10-30T18:03:39.529 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91165.log 2023-10-30T18:03:39.530 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.157981.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157981.log.gz 2023-10-30T18:03:39.530 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131356.log 2023-10-30T18:03:39.530 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91165.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91165.log.gz 2023-10-30T18:03:39.531 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T18:03:39.532 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.131356.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.131356.log.gz 2023-10-30T18:03:39.602 INFO:teuthology.orchestra.run.smithi177.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T18:03:39.805 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mon.c.log: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.91745.log.gz 2023-10-30T18:03:40.715 INFO:teuthology.orchestra.run.smithi170.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.91700.log.gz 2023-10-30T18:03:40.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-10-30T18:03:40.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-10-30T18:03:41.105 INFO:teuthology.orchestra.run.smithi177.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T18:03:41.145 INFO:teuthology.orchestra.run.smithi170.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T18:03:41.905 INFO:teuthology.orchestra.run.smithi170.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T18:03:42.989 INFO:teuthology.orchestra.run.smithi177.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-30T18:03:43.005 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T18:03:43.221 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2023-10-30T18:03:43.255 INFO:teuthology.orchestra.run.smithi177.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T18:03:43.273 INFO:teuthology.orchestra.run.smithi177.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-30T18:03:43.530 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T18:03:43.582 INFO:teuthology.orchestra.run.smithi177.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T18:03:43.584 INFO:teuthology.orchestra.run.smithi177.stderr: 2023-10-30T18:03:43.584 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m4.274s 2023-10-30T18:03:43.584 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m17.271s 2023-10-30T18:03:43.584 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.598s 2023-10-30T18:03:43.672 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T18:03:43.883 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T18:03:43.885 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-30T18:03:43.885 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m4.613s 2023-10-30T18:03:43.886 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m22.493s 2023-10-30T18:03:43.886 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m0.821s 2023-10-30T18:03:43.886 INFO:tasks.ceph:Archiving logs... 2023-10-30T18:03:43.887 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201/remote/smithi170/log 2023-10-30T18:03:43.888 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T18:03:45.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~5s 2023-10-30T18:03:46.936 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201/remote/smithi177/log 2023-10-30T18:03:46.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T18:03:49.419 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 328, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-10-30T18:03:49.420 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T18:03:49.432 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 328, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 1471, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_fee4af45ddecf08e014b4503e218408d9e5c5be2/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-10-30T18:03:49.432 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T18:03:49.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T18:03:49.436 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T18:03:49.517 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T18:03:49.529 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T18:03:49.529 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T18:03:49.535 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T18:03:49.548 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2023-10-30T18:03:49.551 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2023-10-30T18:03:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T18:03:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-10-30T18:03:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 +860us[ +860us] +/- 83ms 2023-10-30T18:03:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -211us[ -172us] +/- 62ms 2023-10-30T18:03:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 12 -272us[ -272us] +/- 62ms 2023-10-30T18:03:49.554 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T18:03:49.556 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T18:03:49.556 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2023-10-30T18:03:49.556 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +830us[ +830us] +/- 83ms 2023-10-30T18:03:49.556 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -275us[ -280us] +/- 62ms 2023-10-30T18:03:49.556 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -343us[ -348us] +/- 62ms 2023-10-30T18:03:49.557 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T18:03:49.558 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T18:03:49.569 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T18:03:49.570 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T18:03:49.616 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-30T18:03:49.718 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-30T18:03:49.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T18:03:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T18:03:49.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T18:03:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T18:03:49.885 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T18:03:49.897 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T18:03:49.943 INFO:teuthology.task.internal:Duration was 2877.104776 seconds 2023-10-30T18:03:49.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T18:03:49.992 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T18:03:49.993 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T18:03:49.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T18:03:50.039 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T18:03:50.042 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T18:03:50.370 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T18:03:50.371 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2023-10-30T18:03:50.371 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T18:03:50.391 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2023-10-30T18:03:50.392 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T18:03:50.419 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T18:03:50.419 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T18:03:50.435 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T18:03:50.488 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T18:03:50.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T18:03:50.502 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T18:03:50.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T18:03:50.636 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T18:03:50.637 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T18:03:50.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T18:03:50.691 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T18:03:50.703 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T18:03:50.706 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T18:03:50.729 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2023-10-30T18:03:50.756 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2023-10-30T18:03:50.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~10s 2023-10-30T18:03:50.777 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T18:03:50.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T18:03:50.798 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T18:03:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T18:03:50.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T18:03:50.844 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T18:03:50.845 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201/remote/smithi170 2023-10-30T18:03:50.846 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T18:03:50.882 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201/remote/smithi177 2023-10-30T18:03:50.883 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T18:03:50.919 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T18:03:50.919 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T18:03:50.922 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T18:03:50.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T18:03:50.983 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T18:03:50.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T18:03:51.026 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T18:03:51.026 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T18:03:51.029 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T18:03:51.042 INFO:teuthology.orchestra.run.smithi170.stdout: 265243 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 18:03 /home/ubuntu/cephtest 2023-10-30T18:03:51.043 INFO:teuthology.orchestra.run.smithi177.stdout: 265781 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 18:03 /home/ubuntu/cephtest 2023-10-30T18:03:51.045 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T18:03:51.076 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T18:03:51.126 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2023-10-30T18:03:51.127 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T18:03:51.152 DEBUG:teuthology.nuke:shortname: smithi170 2023-10-30T18:03:51.152 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T18:03:51.173 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 17:07:24.953494', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEsA2T3Z7BhEScoWd2R+zeQ3a4mkh6Qpyg5hRdYQ+wKe/dOBtKMtrJRAdobB9gMKKCdoByietgpKaqqfGE15ufg='} 2023-10-30T18:03:51.179 DEBUG:teuthology.nuke:shortname: smithi177 2023-10-30T18:03:51.179 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T18:03:51.199 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 17:07:24.951707', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKldg/iMffn8uQIfGj3LsSXEf/wPuI0S90sdsB603K8/X9pi5GwOwl1hilfS5DV06t4Vp5SxbwhF17bGKJ+zSg='} 2023-10-30T18:03:51.223 INFO:teuthology.orchestra.console.smithi170:Power off 2023-10-30T18:03:51.224 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T18:03:51.246 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2023-10-30T18:03:51.247 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T18:03:51.261 INFO:teuthology.orchestra.console.smithi177:Power off 2023-10-30T18:03:51.261 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T18:03:51.270 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-10-30T18:03:51.283 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2023-10-30T18:03:51.283 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T18:03:51.307 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2023-10-30T18:03:55.274 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T18:03:55.308 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T18:03:55.384 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-10-30T18:03:55.418 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2023-10-30T18:03:55.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~15s 2023-10-30T18:03:59.388 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T18:03:59.419 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T18:03:59.499 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2023-10-30T18:03:59.531 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2023-10-30T18:03:59.601 INFO:teuthology.orchestra.console.smithi170:Power off completed 2023-10-30T18:04:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~20s 2023-10-30T18:04:03.534 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T18:04:03.645 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2023-10-30T18:04:03.746 INFO:teuthology.orchestra.console.smithi177:Power off completed 2023-10-30T18:04:03.850 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 2877.1047763824463 failure_reason: saw valgrind issues flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-10-30T18:04:03.850 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T18:04:03.937 INFO:teuthology.run:FAIL